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Instructional Systea Developaent USD) sethodoldgles 
and practices were exaained J#n,the Aray, Navy, Marine^ Corps, and Air 
Force, each of which prescribes the ISD systea involving rigorous 
derivation of training requireaents fron job reguireaents, selection 
of instructional strategies to aaxiaize training efficiency-, and 
.revision of instruction until training objectives are net. Procedures 
included (1) analysis of- quidance documents used in all services, for 
conducting ISD, *2) questionnaire survey of 2^09 units, agencies, and 
, schools where training is developed,/ and (3) interview of training 
developers at thirty-three organizations to deteralne how fifty-seven 
courses were designed* The najor finding was that ISD is not being* 
used. The iterative and derivative character of the ISD eethodology 
.can insure that training will be relevant to job reguireaents if its 
(procedares are carried out. In practice, however, nan y ISD coa'ponents 
Jare oaitted so that the process is not truly derivative. The eleaents 
of ISD actually being practiced involve aodern training technology 
for job analysis and self-paced instruction. Evidence shows that 
although the products of ISD can* be aandated the ISD process itself 
cannot because training is both developed and evaluated within the 
training subsystea, whereas the consequences of training occur in 
operational units. It is thus recosnended that operational connands 
be given a larger role in identifying job 
training reguireaents, and evaluating the 
graduates. (Beconsendations are presented 
in the ISD process. ) (JT) 
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Mar 1979. Findings are based on (1) analysis of the primary guidance docu-* 
ments used in the Armed Services for conducting ISD, (2) questionnaire survey 
of 209 units, agencies and schools where training Is developed in the Ser- 
vices, and (3) detailed interview of training developers at 3^ organization*, 
to determine how 57 courses, were designed. Major findings are that ISD is 
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20. Abstract (continued) 

not being used cither to optimize total system effectiveness or to maxi- 
mize training efficiency* 

The iterative and derivative, character of the methodology can insure that 
training will be relevant to job requirements if its procedures are faith- 
fully carried out. In practice, however, many of the Components of ' SD 
axe otaitted and thte close connection between components to make the process 
truly derivative is not maintained. Thus the potential of ISD to insure 
that* training m*ets job requirements is not being Tealized. The conception 
of ISD that is most adequately represented in current applications of ISD 
is as the use of specific elfemerits tff modern ttaihing technology, i.e. , jtfb 
analysis, self-paced instruction. Considerable evidence leads to the con- 
clusion that although the generation of the products of ISD' can be man- 
dated, the ISD process itself cannot. Training is bqth developed and • 
evaluated within the training subsystem, whereas the consequents of train- 
ing occur in operational units. It is reccunmended that operational com- 
mands W given a* larger role in identifying job requirements, establishing 
training. requirements, and evaluating the performance of training graduates. 
The report also contains findings and recommendations forl9 specif ic* steps 
of the ISD process. . % - 
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PREFACE > 
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• * « 

This report is an -^analysis by tho / Human Resources Research Organization of Instruc- 
tional System Development (ISD) methodology and of practices observed in its application 
in the Army, Navy, Marine Corps, and Air Force. The research was conducted under 
Contract No. MDA 903-77-C-0335 for the Office of the Assistant Secretary ot Defense 
for Manpower, Reserve Affairs and Logistics. The work was monitored at different times 
by Colonel George P. Tilson, Ms. Jeannie Fites, and Major Ken Rousseau, all of OASD 
(MRA&L). Their assistance is gratefully acknowledged. 

The research was performed by HumRRO Western Division at Carmel, California, 
Dr. John.E. Taylor, Director. The Principal Investigator was Dr. Robert Vineberg; 
Research Associate was Mr. John N. Joyner, who contributed to all phases of the study 
and writing of the final report. 

The authors wish to acknowledge the cooperation and assistance of the many 
persons-officers, enlisted men, and civilians— who provided information about the use 
of ISD in the military services. Their number precludes mention by name. However, 
special mention should be made of these persons who arranged for the distribution of 
the JSD survey questionnaire: Army - Dr. Joseph Kanner, Training and Doctrine Com- 
mand; Navy - Dr. Worth Scanlarid, Office of the Chief of Naval Education and Training, 
Commander Joe Funaro, Naval Training Equipment Center, and Commander Paul 
Chatelier, Naval' Air Systems Command; Marine Corp: - Lt. Col. E.A. Grimm, Head- 
quarters, Marine Corps; Air Force - Maj. George Hittle, Air Staff Operations and 
Readiness. ' ' ; 



in 



SUMMARY AND RECOMMENDATIONS 



PROBLEM 

The Army, Navy, Marine Corps, and Air Force all prescribe a similar sequence of 
procedures for the development of training. These procedures, called Instructional Sys- 
tem Development (ISD), are characterized by: 

(1) Rigorous derivation of training requirements from job requirements. 

' Training requirements are tfi be selected so as to maximize the combined 
effectiveness of the training" and noiv-training components of a total 
operational system. 

(2) Selection of instructional strategies to maximize the efficiency of training. 

(3) operative tria ^ anc * revision of instruction during development until training 
objectives are met. 

ISD, a systems approach to training development, has many potential advantages, 
but It is demanding to carry out. It requires sustained commitment to a repetitive 
process of analysis, design, verification, and revision. Experience in attempts to insti- 
tutionalize such a process has tended to reveal problems. 

This study, performed for the Office of the Assistant Secretary of Defense (MRA&L), 
examined ISD methodologies and practices in the four Services to seek answers td the 
questions: / 

• Do current methodologies as represented ^n the major guidance documents 
useci in the Services provide the means foi attaining the gqals of ISD? 

• Do current applications of ISD reflect these goals? 

• How can ISD methodologies and applications be made more effective? 

• ■ • 

• • • 9 

APPROACH . v . 

• \ 9 ' , ° • 

The .absence of reliable criteria of system performance precludes any attepipt to 
evaluate the effects of ISD on performance of the total operational military system. The 
study, focuses instead on the training subsystem, and within^this subsystem it 4s restricted 
primarily to an an^ysis of methodology, proeess, and intermediate products as they bear 
on ISD. 

The analysis is ttased uffbji information developed in three major activities: 

• Analysis oMhe primary documents currently in use in the Army, Navy, 
Marine Corps, and Air Force for guidance in conducting ISD. 

— Interservice Procedures for Instructional Systems Development (1975), * 
published by the Army as TRADOC Pamphlet 350-30, and by the Navy 
as NAVEDTRA 106 A. Developed 'for the Army and later approved by 
fhe Interservfce Committee on Instructional Systems Development. Con- 
tains detailed procedures for conducting ISD. Known as the ITRt) model. 

f - Marine Corps Order PI 518.23B ( 1978). A greatly reduced version of , the 
ITRO model, incorporating documentation requirements specific to the 
Marine Corps. . " . ' 
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Ait, Force Pamphlet 50-58, Handbook for Designers of Instructional 
Systems (1973-1974). Detailed procedures for implementing the con- 
cepts of AF Manual 50-2, an overview of the ISD process. 

• .. 

• Questionnaire survey of 209 units, agencies, *nd schools where instruction 
is developed in the four Services. 

• Detailed interview of training deyelopers at 33 of the organizations surveyed 
to determine how 57 specimen courses were designed! * 

A generalized ISD model composed of 19 training development steps was used to 
provide a structure for analyzing and organizing the information obtained, and summarizing 
findings about ISD practices. The steps and requirements of this generalized ISQ model 
were derived in part from synthesizing the Service models, and in part through rational 
analyses of the intended functions of those models. 



GENERAL FINDINGS AND RECOMMENDATIONS 

* ♦ • 

Judgments about the adequacy of Instructional System Development, its representa- 
tion in the Service models, and the way it is currently, being applied depend on one's I 
conception of what ISD is and what it is expected to accomplish. , 

The most general way to define ISD is. as a means of designing training to optimize 
total system effectiveness. ' Criteria do not exist for measuring total system effectiveness; 
ISD is not being used to achieve, nor can it be expected to result in, optimizing the 
effectiveness of the total system. 

A more* circumscribed view of ISD is as a methodology for maximizing training 
efficiency within the training subsystem. However, information about the effectiveness 
and costs of different training strategies is far, Trom complete, and a trial -and -error 
approach to maximizing efficiency is not practical, given the number of possible' combi- 
nations of methods. While ISD does provide a framework for comparing alternative 
training strategies, it is not currently being used, nor can it be regarded, as a methodology 
for maximizing training efficiency. 

A more highly focused view is that ISD is a methodology for insuring that training 
is relevant to the job. Its iterative and derivative character virtually assures that training 
will be relevantrtf-available procedures are faithfully carried out. In practice, however, 
many of its components are omitted, and the close' connection between components that 
is essential to make the process'truly derivative is not maintained. Most important, the 
testing and revision necessary to insure job relevance generally do not occur. The potential 
of ISD to insure that training meets job requirements' is not being, realized. 

A final conception of,ISD is that it is synonymous with the use of modern training 
technology, i Any of the' steps in the training development process are a part of modern 
training technology,' and so are any of those particular training and evaluation methods 
currently being emphasized (e.g. ^self-paced instruction, computer-managed instruction,, 
criterion-referenced testing). This definittorPof ISD is clearly the least demanding, since 
in essence it holds that undertaking any training development step or using any such * J 
training or evaluation method constitutes ISD. It i3 the definition that is most adequately 
•represented in current applications of ISD. 



In summary, then, two effects of ISD are currently possible: insuring that training * 
meets job requirements and promoting the use of modern training technology. The former, 
which is clearly the more desirable, is not being achieved. The ISD modes does provide 
the methodology for making training relevant, but tfce mere existence of the model does 
not compel trainers to follow it. Trainers are relatively free, within fairly broad limits, 
to determine the extent to which they will conform to the ISD process and actually* use 
its products in designing trainiog. Forexample, front-end analysis may be undertaken,, 
but its results can be and frequently are ignored. Training objectives often are devel- 
oped with indifference to, or in ignorance of /actual task requirements. Many tests and 
instructional materials are developed without regard to training objectives. Information^ ^ 
about the performance of trainees, once collected, is often npt used to revise training, . 
and feedback about graduates, if initiated, is often, not acted upon. 

Because ISD is a process, it is difficult to observe directly. The occurrence of a 
process is generally inferred from the presence or absence of its products. In the case 
of ISD, however, the mere existence^of its products— job task lists, training objectives, 
achievement tests, and the like-does not by itself indicate that they have been used in 
training development. Considerable evidence that many ISD products remain unused 
le^ds to the conclusjton that, while the generation of these products can be maudated, 
the ISD process andVthe appropriate use of the products during training development 
cannot. i % 

Similarly, a routine allocation of responsibilities in ISD does not necessarily guar* 
antee that these responsibilities will be recognized', accepted, and carried out. For example, 
ISD methodology requires developers t^ specify the way in which proficiency will be 
developed or mediated for all job tasks— immediately, through entry training; later, 
through advanced traimrtt or job experience; or through direct support of job performance 
by procedural aids or other means. Yet this requirement is frequently ignored. ISD is / 
generally conducted to develop/training for only one particular setting, and the manner in 
which s(cill will be acquired fot tasks that are not selected for training in that setting is 
usuqjly not specified. Even wHere skills were explicitly identified for later acquisition* 
the present study found little evidence either that means were developed for subsequent 
training v or that operational units were informed of their responsibilities for insuring that 
these skills b**acquired. (This observation is based on training development within the 
training subsystem. The present study did not examine training developed or conducted 
in operational units. Possibly such an examination would indicate that training" for job 
skills in these units is being conducted in a more comprehensive manner than was sug- 
gested by the evidence in the present study > 

The current failure pf ISD applications to insure that training meets job requirements; 
then, is largely due not to inadequacies, in the methodology, but to omissions and to * 
failure to use its products in a way that makes the procesytruly derivative. An implica- 
tion of thes^ findings is that future efforts to implement ISD should concentrate on 
finding ways -to maintain the integrity of the model. 

The findings of the study do not of themselves indicate how to assure rigorous 
adherence to what is clearly a very demanding model, byt they do suggest that intro- 
ducing changes solely within the training subsystem is not likely to have any great feffect. 
The data strongly suggest the need for checks and balances to guard against omissions in 
the ISD process and failures to use the ISD products. It wpuld appear logical, therefore, 
to provide for an expanded role by operational commands— the party directly affected 
by shortcomings in training, and best -able to assess the effects of training. 



Such an expanded role for operational commands is, in fact, implied by the model 
itself. The derivative and iterative aspects of ISD depend on feedback and exchange of 
information between trainers and users. It is difficult to see how ISD could be more^ 
rigorously applied unless such an exchange takes place. In principle, the training sub- 
system seeks"informaiion about field requirements ana performance' as the foundation on 
which training is constructed. In practice, however, the study found that the training 
subsystem does not have this basic orientation; often giving insufficient attention to the 
effective. use of this information. It . is reasonable to suggest that,a balanced relationship- 
one that fosters active participation by, and cc^ftpcmnication between, operational and 
training commands— is essential to the ISD processT 

The following recommendations define the means by which operational commands 
can assume a "greater role. Under these recommendations, operational commands would, 
participate to the greatest extent in those parts of ISD where job performance is repre- 
sented and where job performance requirements are translated into training requirements. 
Those parts of ISD concerned with the design pf Instructional strategies to meet train- 
ing and job requirements would remain the province of the training community. 

It is recommended: . V 

(1) That job requirements (skilft and knowledge required fo^ succes&ful job 
performance) be jbintly defined, and agreedto, by training and opera- 
tional commands. This recommendation is a prerequisite to Recommenda- 
tion 2; and to all subsequent training development activities. If successful 
dialogue, negotiation, and agreement are to follow, training requirements 

trust be based on a mutually agreed-upon definition of job requirements, 
♦ * * * 

(2) That training requirpmenti (skills and knowledge to be available at the 

conclusion of training) be jointly defined* and agreed to, by training 
and operational commands. " Such an agreement should identify the specific 
tasks and standards to which proficiency will be developed, and should 
delineate the respective responsibilities of the two parties, -This includes 
providing a means for.bringing job incumbents'to the desired level of 
proficiency whenever agreed-upon training requirements do not match job 
requirements. * 

{$)' That operational commands be required to evaluate the performance of 
training graduates, and report their findings to the training commands. 
Unless operational commands evaluate performance, feedback fnom users 
to trainers will not have a sound basis. Without reliable information about 
the effects of training, specification of training requirements will not 
serve its purpose. 

(4) To implement Recommendation 3, that task-specific criteria for evaluatin { 
the performance of graduates, including methods and*standards to be 
employed, be jointly defined, and agreed to\ by training and operational 
commands. Evaluation criteria should be at the task level of specificity 
to permit clear and useful diagnosis of training. More general evaluations 
are of little use in isolating the causes of inadequate performance. Eval- 
uation criteria must be jointly agreed to if the results of evaluations ale 
to be accepted as valid, and acted upon. y „ 
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' r Giving the operational command a greater role in both establishing training require- 
ments and determining whether requirements have been met will not of itself guarantee 
that ISD procedures will be rigorously applied to the development of training. It will, 
however, increase theMnvolvemcnt *of those who have the most fundamental interest in 
seeing that training has been adequately designed and conducted. * 

So long as training development and evaluation are regarded as a separate activity 
of the training commands, there is little reasonto expect that ISD°will be applied any 
more effectively than under the present conditions. If training and operational commahds 
share these activities— each making its specialized contribution to complement the wojk * 
'of the otherrthe potential of the ISD process for improving training will be enhanced. 

FINDINGS AND RECOMMENDATIONS FOR . 
SPECIFIC ISD STEPS/ 

. f Need/Discrepancy Analysis 

•Findings m 

Methpdolo gy - Present ISD models go not include procedures for identifying problem! 
in existing training and for identifying a need to undertake ISD. Guidance is needed 
on how to identify 'discrepancies between existing" training and the field reqqirements 
for a job, and how to revise training shbrt of undertaking the entireJSD process. 

2, Application - ISD is not generally initiated in response to specific discrepancies 
betweerr-uaining and field requirements. 

3, Application - iSD is usually initiated in response to a directed change (e.g., 
provide individualized and self-paced instruction) or to a requirement to revise 
existing framing in accordance with ISD methods. 

Recommendations ■ ^ 

1. Where existing training is being examined, ISDpitlVthodologies should emphasize * 
evaluating and -improving the training, 0 rather than simply assuming that 'development 
of a new course is appropriate. At present new courses are' rarely evaluated any 
jnore rigorously than the ones they replace. 

2, To increase the emphasis on evaluation, and improvement, specific procedures should 
be developed both to identify faults in existing training and to determine efficient • 
boundaries for the ISD process. P 



IDENTIFICATION OF JOB REQUIREMENTS V 

Findings • * ♦ 

1. Methodology - Current- guidance for identifying job requirements permits con- 
siderable latitude in the approach taken and the level of description used, with 
resulting variation in the reliability and utility' of the information developed. 
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2. Application Job analysis is usually confounded with* the selection of tasks for 
training. Emphasis is no sr/en to independently specifying requirements a* they 
exist in the job. 4 

. 3. Application - Job task lists from occupational surveys (e.g., CODAP, NOTAP) are 
sometimes available. The information provided by such lists, however, is often in 
part about classes of activities rather than about tasks, and to that extent irpy not 
serve as an adequate base for deriving training. Often, complete task lists are not 
developed. ^ • * 

Recommendation ^ 

* • ** * 

1. Training developers should be required to provide and maintain a description of job 
task requirements distinct from a listing of tasks selected for training. This tooiild 
make explicit the extent to which trairiThg requirements differ from job requirements, 

.Selection of Tasks for Training 

Findings 

1. Methodology - There are no measures of system effectiveness that can be used to 
validate criteria for selecting tasks for training, and rules for applying such criteria. 
As a result, the chjdjcfe of criteria to be used in selecting tasks to be trained is left 
to personal judgment. 

■ * *• • 

2. Application - Selectiitt tasks for trainipg is generally not preceded by a separate 

and distinct deline^tifti of the tasks required by the job. 

3. Apfrlicatioft - Task selection for training is usually Hot done systematically. Its 

rationale is rarely explicit. 9 

4. Application - Task priorities (that would provide the basis for getting the maximum ^ 
training benefits from the available funding) are not specified. 

Recommendations , 

1. Training developers should be required to make explicit the basis on which they 
select tasks for training, and to specify priorities among both the tasks selected and 
the tasks rejected for training for a particular job. 

2. In the absence of information about the effects of task selection criteria on system ' 
performance, guidance should be devel6ped on the types of task priority information 
likely to tie relevant for different classes of jobs. 



Analysis of Tasks 

Findings * 

1. Methodology * Procedures for analyzing tasks are adequate. 

2. Application - Job task conditions and standards, as distinct from training -conditions 
and standards, are seldom identified. Course controf "documents do not require that 
job characteristics be specified. 



Recommendation 

yi. Benefits of attempting to modify current practices do not appear «reat relative to 
costs. No change in present practice is recommended. 

■ ' , y . . • 

Construction of Job Performance Measures , 

Findings 

1. Methodology - Procedures for developing JPMs are adequate in the ITRO model, 
unclear in the Marine Corps model, and not includad in the AFP 50-53 model. 

2. Application - JPMs have been developed in jobs where the consequences of inade- 
quate performance are especially serious, such as' in flight training or use of special* 
weapons. 

3. Application - In only one instance were JPMs developed as part of ISD to Validate 
within-course tests. * / 

Recommendation 

1. ' While theoretically worthwhile, JPMs are costly to develop. It is unrealistic to 
recommend their development except in special instances. No change in present 
practice is recommended. " — 



Selection bf Setting 

Findings 

1. Methodology - A systematic procedure for determining "the optimal setting for 
training does not exist. The^development of such a procedure waits upon a means 
for measuring systemupefformance, which in turn will permit the validation of site 
selection criteria and decision rules. . 

2. Application - In general, training developers do not have the authority to designate 
>nd develop training in different settings. ( 

Recommendation 

V 

1. The choice o.* training setting has effects on the operational system beyond the 
training subsystem. It is appropriate that the setting selection be made at a higher 
level than training developer. In the absence of a means for assessing total system 
effectiveness, no change in present practice is recommended. 

Development of Training 
Objectives and Objectives Hierarchies 

Findings , . - 

1. Methodology - The ITRO model provides the most comprehensive, explicit, and 
straightforward procedures for translating job requirements into training.objectives. 
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Deficiencies in the Marine Corps model could encourage writing objectives to match 
what is bieing taught, rather than to meet job requirements. 

2. Application » The specification of training objectives id virtually" universal, but the 
procedure^ used to identify objectives are highly variable and frequently unclear. 
There is eyidence that objectives are often prepared after*the fact and are derived 
from training content rather than used to generate it. \" • 

3. Application - Terminal objectives are seldom subjected to an explicit hierarchical 
analysis to derive intermediate objectives* A determination of the skills and knowl- 
edge that would enable the trainee to meetffce terminal objectives would emerge 
frpm such an analysis; these often are not identified. 

4. Application - Even where records are maintained, formats for displaying the relation 
between tasks and training objectives make it hard to determine what objectives, 
have been derived from a given task. Jhat is, tasks that represent objectives are 
displayed, rather than objectives that have been derived from each task. TJius, the 
justification for training for specific objectives is often not clear. 

Recommendation ,' s 

The derivation of training objectives from job tasks should: be made explicit in a 
format that ^ross-references objectives by task. 



Development of Achievement Tests 
Findings * 

\. Methodology - All of the models require that achievement testB^be developed from 
Vaining objectives, rather than from the content of lessons, and all provide some 
information about test construction. All models lack procedures for -maintaining 
congruence between the t&havibrs implied in an objective and the actual require- 
ments imposed by test items (e.g., use performance tests to measure skilled behavior^ 
require that concepts be applied when an objective implies their use rather than 
their recall or recognition). , ; 

2. Application - Many achievement tests are derived, not directly from training objec- 
tives, but from training content. .Knowledge tests &re particularly likely to be 
derived farom content. In these cases, no independent criterion exists to deter- 
mine whether training objectives have been met. • 

3. Application - In general, little or no consideration is given to matching the type and 
level of test items to the behavioral requirements of objectives (see Finding 1). 

Recommendations 

1. ISD models should explain and emphasize the purpose and need for deriving 
achievement tests from training objectives rather than from training content. , 

2. ISD models should be expanded to provide procedures for identifying and main- 
taining congruence between the behavioral requirements of objectives and test items. 
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' Identification of Entry Behavior 

Findings 

1. » Methodology • The major ISD models provide procedures for adjusting training 

objectiyes to match trainee capabilities. 

2. Application • Estimates of capabilities of trainee populations are not verified by ' 
t testing before the training is implemented. 

Recommendation 

•"*#*,. 

1. The advantages of correcting inaccurate estimates of trainee capabilities do not 
appear to justify the costs of the measurement that would be required. Over- 
estimates are likely to be identified during the validation of instruction, and under- 
estimates usually become evident during the conduct of instruction. No change in 
present practice is recommended unless large investments in instructional materials 
are involved (e.g., Training Extension Coijrse (TEC) development). 

Classification of Objectives and 
Selection of Instructional Activities 

Findings 

1. Methodology - f rocedures for classifying training objectives, and for selecting instruc- 
tional activities accordingly, are not highly developed. Different models use different 
taxonomies for classifying objectives, and guidance for both classifying objectives 
and selectingjnstructional activities is provided largely by example rather than by 
means of explicit decision rules. 

2. . Application - Training objectives generally are not classified, and instructional activ- 

ities generally are not specified. 

"V 

Recommendation 

1. Explicit decision rules for classifying objectives and selecting instructional activities 
should be developed. In the absence of such rules, no change in present practice 
is recommended. 

\ / 

\ ' • • 

Selection of Instructional Methods 

Findings 

1. Methodology - All models describe- and discuss alternative instructional methods. 
They specify prerequisite conditions (e.g., setting, group size) for the use of particu- 
lar methods but provide little information about the relative effectiveness of different 
methods, either for particular types of content or for trainee populations. Though 
perhaps sufficient to allow the training developer to reject inappropriate methods, 
the information base and the models themselves are not sufficient to provide for 
selecting optimal methods. 
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2. Application - Training methods are not systematically selected either on the basis 
of instructional activities fwhich are also not specified) or on the basis of trainee 
characteristics. x * 

3. Application - Changes in training methods are almost always in response to com- 
mand policy. 

Recommendation 

1. Present training technology is not advanced enough to support proceduralizeSk 
derivation of training methods from previously specified instructional activities 
and trainee characteristics. Information about optimal training methods for dif- 
ferent training situations, and procedures to enable developers to identify the most 
promising riiethods, should be developed. In the absence of such procedures, no 
change in present practice is recommended. 



Selection of Media 
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Findings 

* - * 

1. - Methodology - Procedures fot selecting training media Appear adequate to match 

' the presentation (stimulus and response) requirements of instructional activities to 
v . appropriate media, if .instructional activities have been specified in sufficient detail. 

2. Application - Training media are not systematically selected on the basis of require-, 
ments of instructional activities (which are also not specified). 

3. Application - Developers generally do not H&ve freedom to select among alternative 
media. Choices and changes in media are usually directed by command policy. 

Recommendation 

1. Selection of appropriate media is contingent on how well instructional activities 
have been specified. No change in the current models for matching media to 
activities is appropriate until activities are rtiore widely specified, and these* models 
can be tested. ♦ 

• ^ . v 

Grouping and Sequencing of Instruction 

Findings ' ' *' 

1. Methodology - Universal principles for grouping and sequencing instructional objec- 
tives do not exist, other than that dependent objectives be placed later than those 
on which they depend. Alternate theories and conflicting strategies abound, with , 
no knowledge base* for resolving them. Because systematically related principles 
for promoting learning are lacking, sequencing instruction must be left to the 
individual judgment of the training developers. 

2. Ap plication - Practic%ln grouping and sequencing instruction generally give no con- 
sideration to learning requirements other than the orrler imposed by dbvious depend- 
encies. Attention is given to constraints of non-lea r ning factors' such as. equipment 
availability and scheduling. 



Recommendation 

1. Organized and systematically related principles of learning on which to base grouping 
and sequencing decisions are not available. The development of such information 
falls within the psychology of learning and is beyond the scope of recommendations 
in this report. In the absence of such information, no change in present practices 
* is recommended. 



Development of Plan for 
Authoring and Managing Instruction 

/ 

Findings 

1. Methodology • All models require ^ie* preparation of a plan for authoring and 
managing instruction. » 

2. Application - Plans that record course content are often prepdfed. 

3. Application ^ Plans typically do not specify instructional events and are rarely used 
t6 develop instructional materials., v 

Recommendation 

1. Explicit decision rules for selecting instructional activities do not presently exist 
(see Classification of Objectives- and Selection of Instructional Activities above). 
Until such rules are available and generally acknowledged as valid, a requirement to 
specify instructional activities in an authoring and managing ^)lan is likely to be 
viewed as a pointless exercise. No change in present practice is recommended. 

Review and Selection of Existing Materials 

Findings 

1. Methodology - The ITRO and Marine Corps modejs state that decisions to use 
Existing materials (rather than author new instruction) are to be based on the 
appropriateness of these materials to the previously specified characteristics of 
objectives, methods, and media. The AFP 50-58 mode 1 does not provide guidance 
for reviewing and selecting existing* material. 

2. Application - Characteristics to be identified in judging the appropriateness of 
existing training materials are not specified. 

3. Application * Review and use of existing training materials is minimal, except for 
those in a course 1 that is, being revised. 

* n 

Recommendation 

1. The specification of necessary properties of materials for particular training situa- 
tions, and the description and cataloging of existing materials to permit the inter- 
change of matching components across courses, represent a degree of perfection 
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that is not presently attainable. An attempt to reduce the review, and selection of 
existing materials to a systematic procedure is to act asif the methods of a well- 
developed technology were available in an area in which judgment is in fact the 
dominant factor. No change in present practice is recommended. 



Authoring of Instruction 



Findings 



1. 



2. 



Methodology '- All models specify that instructional materials undergo tryout during 
the authoring process. The ITRO and Marine Corps models emphasize that instruc- 
tion should be lean to insure the economies of minimal instruction. 

Application - There is little awareness of the concept of lean instruction, and 
few attempts to develop it. * 0 ' 

Application - Instruction is rarely given tryout and revision during, authoring. 



/ 



Recommendation 

1. Training managers should receive guidance on the purpose and importance of 
developing lean instruction. Guidance should indicate the role of tryout and 
revision of instruction as a necessary elemeftt of this strategy. , 



Validation of Instruction 



Findings 

1 



Methodology - All models specify satisfactory procedures for validating instruction. 
The adequacy of training Materials for attaining objectives ii verified through the 
administration of achievement tests. ' _ ^ 

2. Application - Validation criteria— that is, evidence that instruction is satisfactory— 
are rarely specified. 

3. > Application - Instruction is rarely validated before it is implemented When vali- 
dation do$s 9ccur, it is training materials (ckg., textbooks,, tape/slide programs) 
that are evaluated; instructor lessons and le&on plans are almost never evaluated. 

Recommendation 

1. Of the three major types of ISD evaluation (validation, internal evaluation, external 
evaluation), validation has the greatest potential for effecting improvements in 
instruction. Once instructional materials have been produced arid instruction has 
been implemented, changes are less likely to be introduced, and new materials are 
more difficult to generate. Validation trials to meet specified criteria should bt 
required before new instruction is approved. * 
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Internal Evaluation 

• ' '' 

Findings » ■ 

1. Methodology - All models specify adequate procedures, for the internal evaluation 
of training. Quality control of the training product is to be maintained through 
the. administration of objective-referenced achievement tests. 

2. Application - Evaluation and .revision of instruction based on needs revealed in 
student performance (product evaluation) are generally not done. ~ 

3. Application - Training design decisions are rarely documented (process evaluation), 
to facilitate redesign when instruction is found to be inadequate. 

Recommendation ' n 

1. Trainers should be required to determine and record trainee performance for -each 
objective. Although absolute standards to identify when training revision is needjd 
are. difficult to establish/the recording of specific trainee performance wbuld prt^ 
vide a desirable prerequisite to any revision. Moreover, it would suggest relative 
standards for the need to revise training. 



• External Evaluation 

Findings 

1. Methodology - the ITRO model prescribes the most reliable and most costly way. 
to measure the adequacy of the instructional design process: administering Jqb 
Performance Measures to gi*duates in the field. It also provides the most guidance 
for isolating causes of performance discrepancies after the external evaluation of 
training. If summary evaluations are to be used, the ITRO and AFP 60-58 models 
specify that information be obtained at the task level of specificity, while the 
Marine Corps model does t not. 

2. . Methodology ■ None of the models specify criteria that should be used to determine 

whether training is to be revised as a result of external evaluation , v nor how to 
arrive at such criteria. Norte tell how good job performance must be to indicate 
that training is acceptable. 

3. Application - The effectiveness of training is virtually never evaluated by the admin- 
istration of Job Performance Measures to job incumbents. 

4. Application - Supervisor summary evaluations of job incumbent performance are 
occasionally obtained But usually are not provided at a. task or training objective 
level of specificity. Even when performance and job requirements information is 
obtained, it is rarely used to redesign training. 



Recommendations 

1. The high cost of administering performance tests to job incumbents and the diffi- 
culty of maintaining the necessary degree of objectivity and standardization' pre- ' 

. , elude their widespread use to evaluate training. No change in present practice is 
recommended. - 

2. Training can be evaluated by gathering information (rather than direct measurement) 
about job performance' and job requirements, at a task and training objective level 
of specificity, far more thoroughly than is presently done. The failure to obtain * 
and use such information is a major shortcoming in current applications of ISD. 
While decision rules for using such information cannot at present be based on 
measures of -total system effectiveness, other means for arriving' at such criteria 

are available. It is recommended 'that operational commands define both the 
specificity of the task description and the level of performance they would be 
willing to use to evaluate the acceptability of job incumbents. These are the 
criteria that supervisors should use to judge (rate) field performance and establish 
> the need for training revision. 
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Chapter 1 • 
INTRODUCTION 

SYSTEMS ANALYSIS AND TRAINING DEVELOPMENT 

• 0 

Beginning in the late 1950s,, systematic procedures began to be applied to the design 
* of training programs in the military services.' Early methodologies, influenced in general 
by operations analysis concepts Of World War II and specifically by the recognition of a 
need for requirements analysis, emphasized empirical determination of the requirements 
of jobs and clear specification of the objectives 4i training. 

During the 1960s and early 1970s, instructional design procedures became more codi- 
fied. Models prescribing specific sequences of training development emerged. To the. 
"front-end" analysis of earlier procedures, these later models added steps f or ^the design 
and* development of instructional content, as well as its implementation and control. 1 

Today the Army, Navy, Marine Corps, and Air Force all prescribe a similar sequence 
of procedures for the development of training. For example, all of the Services require 
that the design of training begin with an ariajysis of job requirements. All prescribe that 
these requirements be stated in terms of capabilities to perform specific tasks, that training 
objectives be derived from these tasks, that instructional content be selects solely to 4 
meet the objectives, and that instructional events be determined by characteristics of the 
content, capabilities of trainees, and level of mastery to be attained. None of, the meth- 
odologies permit a rearrangement of the process so as to, for example, allow objectives 
to be determined by already existing instructional materials. 

. These procedures^ though differing somewhat in organization and detail both across 
/and within Services, are all models of essentially the same-process and are currently 
referred to as Instructional System Development (ISD). 2 The process is characterized by 
a number of features common to systems analysis in general. 

• A system is comprised of subsystems and their interfaces. In ISD, training 
is recognized as a subsystem of a larger operational system and interfaces 
with other subsystems (e.g., weapon system and* job requirements, per- 
sonnel classification and assignment policies). Training design decisions during 
jXe ISD process are made with a view toward optimizing the operation of 
both the total operational system and the training subsystem. , 

✓ 

Earlier versions of these procedures included: Design of Instructional Systems (©IS), Systems 
Approach to Training (SAT), Systems Engineering of Training (SET), and Training Situation Analysis 
(TSA). For a historical review of these methodologies and the development of the systems approach 
to training, see Montemerlp, Melvin D. and Tennyson, Michael E., Instructional Systems DeveloprtierA 
Conceptual Analysis A Comprehensive Bibliography, Technical Report NAVTRAEQUIPCEN IH-257, 
Human Factors Laboratory, Naval Training Equipment Center, February 1976. 

The ISD models in use in the Armed Services are described in a variety of -documents and 
manuals. Primary documents currently include Air Force Manual 50-2, Instructional System Development 
(1970); Air Fore* Pamphlet §0-58, Handbdok for Designers of Instructional Systems (1973, 1974); 
Marine Corps Order P1510.23B (1978); Interservice Proced\i#s for Instructional Systems Development 
(1975) (published by the £rmy as TRADOC Pamphtet 350-30 and by the Navjf*a* NAVEDTRA lX)6A). 



• Subsystem objectives and. configurations are based oh an analysis of system 
requirements and subsystem relationships. The ISD process begins with a 
detailed analysis of job requirements and a determination of which require- 
ments nre to be met through training. t Decisions abo/it the functions of. 
the 'training are to be made on the basis of costs and effects to the opera- 
tional system, costs to the training subsystem, 'and constraints of time, 
resources, and feasibility. 

• Systems analysis involves the empirical and iterative evaluation of alterna- 
tives. The impact of alternative training configurations on the operational 
system often cannot be clearly anticipated. Also, principles for analyzing 
training requirements and decision rules for selecting and organizing instruc- 
tional content to meet learning goals are not well developed. Training devel- 
opment requires a series of approximations involving test and modification. 

• Evaluation criteria are specified. Within the training subsystem, instructional' 
development is evaluated in terms of specific criteria that have previously 
been specified (i.e., behavioral objectives stated in terms of tasks, standards, 
and "Conditions) and the extent to which costs have been minimized in 

. meeting these criteria (e.g., lean instruction): Evaluation of the adequacy of 
the ISD process itself (in contrast to the training it has genaWted) must be 
accomplished in terms of criteria external- to the training subsystem. This 
poses a special problem in that criterion measures of the effectiveness of 
the operational system are generally not available. 
While ISD interfaces with other components of the operational system,-it8 focus is 
clearly upon the training subsystem. For example, although theoretically it would be 
possible, the process does not begin with complete freedom in conceptualizing the role 
of the human being in the operational system, that is, allocating functions to be per- 
formed by man and functions to be performed by hardware. 1 Rather, ISD begins with 
either the identification of a need to develop training for a neq job or the recognition 
of a problem in existing training for an old job. In c either case, job requirements are 
largely fixed by the time ISD and the training developer enter the picture. As we shall 
later see, the first ISD decision that has an impact on the operational system is one in 
which tasks are Selected for training; that is, of all the tasks -tMat must be performed by 
a human being, those for which training will be provided are specified. 

The emergence and codification of the. "ISD process have overlapped with the 
appearance ofa variety of techniques that are currently popular in contemporary train- 
ing technology (e.g., criterion-referenced testing, individualization and self-pacing of 
instruction, audiovisual media, computer-assisted instruction, computer-managed instruc- 
tion). As a consequence, there is sometimes a tendency for such Techniques to be 
considered virtually syifonymous with ISD. Such confusion works to the detriment of 
ISD. ISD, however,- has its roots in systems analysis, and as such is primarily a process 
through which training alternatives are selected. It should not be identified solely with 
the application of specific training strategies or products. <>■ 

•'jThe lnter$ervice Procedure for ISD (Executive Summary) explicitly state* that ISD follows a 
specification of requirements of the weapon, force structure, strategy, and personnel subsystems. AF 
Manual 50-2 states that ISD " . . . identifies the nature and scope, of the human role in the system . . 
but no further attention is given to the concept. AF Pamphlet 50 -*8 t which provides detailed procedures 
for implementing the concepts of AF Manual 50-2, omits this phase. 



SUMMARY MODEL OF ISO 



. A brief description of the components or steps of a generalised ISD model follows. 
It is based on an analysis of the major models now in use in the Services 1 A later 
description discusses alternative placement of some of the steps. 

Components of the model are necessarily depicted' in a -linear'sequence. In practice, 
some of the steps may be conducted concurrently while others must.be done sequentially. 
In keeping with a systems approach, all models emphasize the interrelationships of the 
steps. Adjustments are to be made during successive iterations of the process. 



Nac J/Discrepancy Analysis 

The first step in ISD consists of actions that either reveal or confirm a need to develop 
or revise training. If a need to revise training is identified, the point of entry into ISD 
and the boundaries of the redesign process are indicated. 



Identification of Job Requirements 

' Job requirements are identified in what is clearly the keystone of the ISD process. 
This step consists of (a) developing a list of tasks encountered in the job, (b) verifying the 
list with a representative sample of job incumbents, and (c) gathering task priority informa- 
tion.j(e.g., frequency and difficulty of task performance). 

, Selection of Tasks for Training 

i 

Tasks for which training will be given are selected from those identified earlier during 
job analysis. Decisions are based on task priority information, training resources, and costs. 

/nalysis of Tasks 

Conditions of performance, behavioral elements, and standards of performance 
within the job^enyironment are described for tasks that Jiave been selected for training. 
Task Analysis provides informs A lofr heeded later for developing both Job Performance 
Measures and training objectives. 



Construction of Job Performance Measures 

Job-referenced tests of performance are constructed t6 serve as criteria for assessing 
the capabilities of trainees to meet job requirements. 



1 Several steps included in the Service models that do not directly affect the desi t process 
(Implement Instructional JPlan, Conduct Instruction, Revise System) have been omitted from this 
generalized ISD model and from later discussion of the individual models. 



Selection of Setting 

/ 

rl 

The locus of instruction for tasks seleoted for framing is identified. Decisions are 
based on such factors as cost, characteristics of instructional content, sUident lokd* , A 
fend special constraints (e.g., range requirements). Decisions about on-tne-job training 
options must consider their impact on the force structure. \ 



Development of Training * 
Objectives and Objectives Hierarchies 

Most of the preceding steps have dealVwith job^equirements and job-related decisions. 
This step represents a shift in focus and constitutes the first analytic step in the develop* 
ment of instruction. Objectives ar£ statements of performance requirements, conditions, 
and standards. Jerminal objectives specify the overall behavior /outcome of task perform- 
ance. Intermediate objectives specify the behavioral elements and the basic skills and 
knowledge that mediate terminal objectives. Intermediate objectives are identified by 
constructing objectives hierarchies, that is, multilevel displays of a successive breakdown 

of terminal objectives into their supporting Behavior and mediating skill and knowledge. 

y ■ 

Development of Achievement Tests 

Achievement tests are referenced to instructional objectives. Their form depends on 
the nature of an objective; thus, paper-and-pencil tests are used to assess knowledge, per- ' , 
jftfrmance tests are used to. assess behavidr, and so on. Where objectives are similar to 
job requirements, achievement tests may be similar or identical to the job performance 1 
measures developed earlier. 

' Identification of Entry Behavior t 

r 

m % 

Instructional objectives developed earlier were based on estimates of student capa- ' 
bilities at the time of entry intotrtrairfing. These estimates are now verified tt> determine 
whether trainees possess these* previously presumed capabilities. Objectives are modified 
to make then* consistent with the capabilities of entering trainees. Entry behavior may 
alfeo be assessed later, when actual instruction has begun, to determine whether capa- 
bilities of the entry population have changed over time, and to provide diagnostic infor- 
mation for placement purposes when instruction is individually prescribed. 

Classification of Objectives and 
Selection of Instructional Activities 

Objectives are classified as to type of knowledge and skill. Instructional events 
and conditions are selected in accordance with the characteristics of the^papability to be 
acquired (e.g., intellectual skill, information, motor skill) and the letfefof learning 
required (e.g., familiarization, application). .* < 
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Selection of Instructional Methods 



Instructional methods are the means by which trainees are brought into contact with 
4qstrt*ction. They are defined in terms of the size and location of the instructional group, 
degree of individualization, and type of instructional pacing, tutoring, .and management. 
Methods are selected so as to maximize the. effectiveness and efficiency of instruction, . 
and selection depends upon the particular characteristics of the skills and knowledge to 
be acquired, the nature of the trainee population, She setting, and administrative require- 
ments and constraints. 



Selection of Media 

* 

Media are selected depending on the requirements of instructional activities and 
methods previously specified. Media selected must possess stimulus characteristics and 
response opportunities to support the instructional activities (e.g., texts to communicate 
verbal information, operational equipment or simulators to provide practice in tracking). 
Media muit also be appropriate to the instructional methods (e.g., lecture or film for 
group, instruction, sound/slide for self-paced instruction). From media that meet these 
requirements, final selection is made on the basis of such factors as cost,"*availability, 
production capabilities, size of trainee population, and training setting. 



• - ■ *■ 

Grouping and Sequencing of Instruction . 

Topics are usually grouped on the basis of commonality of subject matter and 
anticipated transfer of learning. Dependent relationships among training objectives 
primarily determine the sequence of instruction. Independent and coordinate objectives 
are sequenced in accordance with such principles as simple to complex, familiar to 
unfamiliar and job order. 

Development of Plan for 
Authoring and Managing Instruction 

A "written plan(s) is prepared to transmit the decisions and specifications determined 
in previous ISD steps to those who will author, conduct, and manage the instructional 
program. The plan includes (a) guidelines and supporting information for the preparation 
of lesson content (e.g., how stimulus material is to be presented and performance 
elicited, types and schedule of testing, trainee literacy characteristics), (b) description of 
student management procedures and instructor roles, and (c) administrative information 
(e.g., class size, equipment use, and facility schedules). 



Review and Selection of Existing Materials 

Before new instructional materials'are produced, existing materials are examined to 
determine whether they can be used or adapted for use. In order to be adopted, existing 
Tnaterials must conform to prior specifications for instructional activities, methods, and 
media. 



Authoring of Instruction 



New instruction, intentionally lean in nature, if developed in accordance with prior 
specifications. As first drafts are prepared, small portions are tried out on persons repre- 
sentative of the entry populatiort^and are modified and augmented as needed. 

\ . 

Validation of Instruction 

In validation, newly developed instruction is evaluated prior to implementation, 
under conditions that closely approximate its intended use. Lessons or an entire course 
is administered to small groups of trainees; objective-referenced achievement t ^sts are 
given to determine the effectiveness of instruction; and training is modified as necessary 
to meet standards. 



Internal Evaluation 

Internal evaluation consists of continuing assessment of the effectiveness of instruc- 
tion when it has been implemented. The principal measure of effectiveness is the 
trainees' performance on objective-referenced achievement tests (product evaluation). To 
aid in identifying any deficiencies that may be revealed, internal evaluation includes 
analysis of how the ISD process was actually carried out (process evaluation). 

/ ■ . 

External Evaluation 

External evaluation occurs when graduates of a training program have reached the 
job. Whereas internal evaluation determines whether trainees-are attaining course objec- 
tives, external evaluation uses an external standard, job performance, to evaluate pro- 
ficiency and the adequacy of the^nstructional process. Information is usually obtained 
by administering questionnaires to graduates and their supervisors; occasionally it is 
obtained through testing or observation of performance. • 

ORiftejN AND PURPOSE OF THE STUDY 

% ' "if 

ISD models and their forerunners have been in use for approximately ten years 1 — 
a long enough period so that a study of how adequate they are, and how they are being 
used, can be undertaken. In late 1977, the Office of Assistant Secretary of Defense 
(MRA&L) requested HumRRO to undertake such a study. The objectives were to 
gather information about the effectiveness of the various ISD methodologies and to 
determine how they* were being applied in each of the Services. What problems had 
arisen in their ^application? How might the procedures be made more effective? 

1 Montemerlo and Tennyson, loc cit., indicate: *The Navy fint entered this field with the devel- 
opment of the Training Situation Analysis (TSA) methodology (Bertin, 1963; VanAlbert, tt at, 1964; 
Chenioff 4 Foltey, 1965). The Army began with Project Minerva, an Army Security Agency study 
which resulted in the De$ign of Instructional Syeteme (DIS) manyal in J966 . . . Two other influen- 
tial manuals of thip period ^ere Butler's (19Cf7) Instructional Sy$tem$ Development (I8D) manual, 
which was written under Job Corps auspices, and Rundquist's course design manual (1966, 1967, 1970) 
which was developed at the Navy Personnel Research and Development Center. 11 




While a systems M> to training has many potential advantages, it is difficult, 
time consuming, and coWfW carry out. It will, be worth the cost and effort only to 
the extent that it is being implemented so as to realize its potential. For example, con- 
siderable data and information from a variety of sources both within and without the 
training subsystem must be developed and analyzed, Interdependences in the develop, 
ment process make coordination necessary among training developers concerned with 
different phases. In each phase there must be accomodation to information and products 
developed in prior phases. The requirement for continuous verification and revision, 
while a major strength of the systems approach, is particularly demanding. To achieve 
the system's goals, instructional systems developers must be able to maintain sustained 
commitment to a repetitive and often difficult process. ^ 

At least one earlier study revealed mariy problems when there was an attempt to 
institutionalize a system* approach to training development. Ricketson, et a/., interviewed 
personnel, who were attempting to implement USCONARC Regulation 350-100-1, Systems 
Engineering of Training - Course Design, at the U.S. Army Aviation School. 1 Even 
though the procedures in the CON ARC Regulation were for less denuding than those 
called for in current ISD models, significant problems werS encounterSd with their inter- 
pretation and application. Guidance was found to be incomplete, contradictory, and 
lacking in detail; individual steps in the process were difficult to relate to one another; 
job analysis data collected in the field omitted' important tasks and described others in 
excessive detail; product review was slow and contributed to wasteful false starts; per- i 
\jonnel turnover was disruptive; and so forth. 

The magnitude of the requirements make it reasonable to ask how ISD is currently . 
working in actual practice. Are the methodologies adequate? Are they being complied 
with? Do they make a difference? 

SCOPE OF THE" STUDY 

As has been indicated, the absence of a criterion of system effectiveness precludes 
evaluating the effects of ISD on the basis of total system performance. Instead, the 
study focuses on the training subsystem and within_thlisubsystem is restricted to an 
analysis of methodology, process, and intermediate products. The job performance of 
course graduates has not been examined. 

The analysis of ISD in this study is based upon information developed in three 
major activities: 

• Analysis of the primary documents currently in use in the Army, Navy, Marine 
Corps, and Air Force for guidance in conducting ISD. 

• Questionnaire survey of 209 units, agencies, and schools where instruction is 
developed in the Services. 

• Detailed interview of training developers from 33 of the organizations surveyed 
to determine how 57 specimen courses were designed. 

Chapter s of the report contains an analysis of the ISD process and of the major ISD 
models being used in the Services. Chapter 3 describes the questionnaire survey undertaken 
to select a sample of ISD applications for detailed analysis. Chapter 4 describes the role and 
structure of organizations performing ISD in each of the Services. Chapter 6 describes how * 
the models are being applied as revealed in the interviews with developers of training. Chap- 
ter 6 summarizes the findings of theitudy and makes recommendations. 

1 RickeUon, Darwin S., Schuh, Ruaael E., and Wright, Robert H. Review of the CONARC Syttemt 
Engineering of Training Program and It$ Implementation at the US Army Aviation School Consulting 
Report, HumRRO, April 1970. 



Chapter 2 

ANALYSIS OF ISD PROCESS AND MAJOR (SD MODELS 

INTRODUCTION , 

„ ■ " . •■ *, 

y It it appropriate to begin, an analysis of Instructional .System Development activities 
/va the Armed Services with an examination both of the process itself and of the models 
and manuals that\are intended ss procedural guides to the process. The issue of whether 
ISD can or cannot be reduced to procedures to be applied by nonprofessional practitioners 
has been discussed, elsewhere. 1 Continuing the debate is not likely to be fruitful: In any 
event the majority of persons who are currently attempting to apply ISD in the Armed 
Services have acquired their understanding of. the process primarily from descriptive manuals 
or from instruction derived from the manuals. It cannot be expected that iSb applications s 
will be any better than the guidance they are based upon. . 

Primarily ISD guidance documents 2 currently In use in the Services are: 

e Interservice Procedures for Instructional Systems Development (1976), pub- 
lished by the Army as TRADOC Pamphlet 350-30, and by the Navy as 
NAVEDJR A 10q A— Developed for the Army and later approved by the Inter- 
service Committee on Instructional Systems Development. Contains detailed 
procedures for conducting ISD. Hereinafter referred to as theTTRO model, 
e Marine Corps Order P1510.23B (1978)-A greatly reduced version of the ITRO 

model incorporating documentation requirements specific to the Marine Corps. 
• Air Force Manual 60-2, Instructional Systems Development <1970, 1976)— An 

overview of the ISD process. ■ 
e Air Force Pamphlet 60-68, Handbo&i for Designers of Instructional Systems 

(1973, 1974)— Detailed procedures for implementing- the concepts in AF 
^Manual 50-2. 
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, 1 Montemerlo arid Tennyson, op. cif./pp. 10*12. 

3 In addition to theea primary documents, various modifications and supplementary documents art 
in use. For example, the U.S. Army Amor School has published the Training Development Handbook 
(1977), which is kr*ely a revision of the ITRO modal thafrdso incorporataa terminology and document*- 
tion requirements specific to tha Army; tha Chiaf of NaTp^Technkal Training has pubUshad CNTT-A10 
Procedure* for the Planning. De*ign f Development, end Management of Navy Technical .Training Count* 

(1976) , spacifying procaduras and documentation formats to ba used in conjunction with tha ITRO 
modal; tha U.S. Naval Health Sciences and Education Command has published tha HSBTC Handbook 
for Implementation (1977), intended to provide supplementary guidance for use of tha ITRO model 
within the command; the Chief of Naval Education and Training has published NAVEDTRA 106 A 
Supplement No. 2, Instructional Program Development, Training Tesk Analy$i$ Procedure* (1978) and 
NAVEDTRA 110, Procedure* for Instructional System* Development (1978),, providing Navy-specific 
guidance for implementing the ITRO model; the Naval Training Equipment Center has sponsored devel- 
opment of Military Specification MIL-T-29063, Training Requirement* for Aviation Weapon Systems 

(1977) , to guide ISD efforts by civilian contractors. These are only examples of the various supporting 
documents that have bean published. 



With the exception of Marine Corps Order P1510.23B, these documents generally 
.are^evailable in each of the Services, One should not assume, for example, that the ITRO 
model is used exclusively in the Army and Navy or that Air Force documents are never 
referred to outside the Air Force. 

The remainder of this chapter contains the step-by-step analysis of the ISD process 
and evaluation of the three major manuals/models: the ITRO model, Marine Corps 
Order P1510 23B, and AJP Pamphlet 50.68. The steps and requirements of a generalised 
ISD model have been derived, in part from synthesizing the Service models and in part 
through rational analysis of their intended functions. Several steps included in the Service 
models that do not directly affect the design process (Implement Instructional Plan,* Con- 
duct Instruction, Revise System) have been omitted from the generalised ISD model. 

Each of the following sections contains a description of a specific ISD step; a summary 
of prerequisite conditions and information, and procedural requirements for performing 
the step; an analysis of the instructions provided in- each of the three Service models for 
meeting the requirements; and a summary commentary. 

STEPS OF PROCESS AND MODELS 

Need/Discrepancy Analysis 



ISD originates with the identification of a need to develop or revise instruction 
Instruction will have to be developed if human performance in* a new job or weapon sys- 
tem needs to be trained; instruction will have to be revised if existing training is not suffi- 
ciently effective and economical. Some models explicitly identify an analysis of needs • 
as part of ISD, while others describe it as an antecedent condition. Whether this step is 
viewed as lying within or without the process is largely a matter of definition. It is impor- 
tant* however, that ISD follow a determination of need rather than be undertaken on a 
prion grounds. The process itself is costly and should be justified, 

prerequisite conditions for determining need are the existence and maintenance of 
procedures for detecting indications of potential discrepancies. In an ideal world, dis- 
crepancies in training would be identified by.cdnstantly monitoring (a) job -requirement* 
(does job analysis indicate that job demands are hot adequately reflected in training or 
that they have changed since an earlier determination?), (b) job performance (does 
measurement of job performance indicate that job incumbents are not adequately trained') 
and (c) course materials (does empirical comparison of alternate types of content and 
media indicate that training is not maximally Cost effective?). These ways of identifying 
discrepancies are, of course, extremely costly. In the real world, these methods must be 
used selectively, so there should be a means to determine when their employment is appro- 
priate. Continuous job analysis, performance testing, and empirical comparisons of alter- 
nate forms of training are neither feasible nor desirable. » ' » 

Indications of changing job requirements and inadequate training may.be as diverse 
as high accident rates, reports of inadequate performance from supervisors and field com- 
manders, and discrepancies between course syllabi and duties reported in occupational 
surveys. Symptoms that training may no{ be effective and efficient are to be seen in 
such indicators -as high course attrition rates, unusual course length and cost, or negative 
comments from studi its and instructors. * , ^ 

When a potential discrepancy is identified, additional information is gathered as needed 
to confirm and define the problem and to* suggest the appropriate locus for corrective 
action, that is, where the JSD process should be entered for remediation. If, for example 
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there is an. apparent discrepancy between jdb requirements and the performance of job 
incumbents, existing training will be examined to determine whether the problem lies in 
an original analysis of^the job, whether errors havfc occurred in the translation of job 
requirements into objectives of training, whether the way in which training is actually 
conducted is ineffective, and so on. The extent of ISD that subsequently will be under* 
taken depends both upon the origin of the problem and upon it* specific characteristics. 
If analysis of the problem reveals a need to add a few new objectives requiring training 
similar to that already given, it is not likely that the entire ISD process will be necessary. 



Summary of Requirements 

Prerequisite Conditions and Information 

1. Information is available about possible discrepancies between training and 

field requirements. ^ 

• . ~~ (T . 

Procedures \ . 

1. ' Analyze initial and Supplementary information. . «• 

2. Identify and specify discrepancy. 1 

3. Specify ISD CTltry point and boundaries of redesign process.) 

Analysis of Service Model* , 4 

JTRO Model, The Executive Summary presents a diagram to show that ISD is pre- 
ceded by an analysis of weapons systems, the force structure, manpower, and existing 
strategies. This analysis, the diagram indicates, leads to, the identification of any "perform- 
ance discrepancy" between requirements on the one hand, and existing programs on the 
other. If no discrepancy exists, of course, nothing need be done. If there is a discrepancy, 
ISD will provide "alternative solutions for selected defined performance discrepancies/* 
The text of the Executive Summary indicates that discrepancies arise from such happen- 
ings as changes in technology or force structure. It suggests an equivalence between this 
type of analysis and the Phase I activities of the ITRO model (job analysis, selection of 
setting), but does not make this explicit. * * <s^_ 

In Step 1.1 (Analyze Job), the ITRQ^ model states that the'first step of ISD is to 
identify the discrepancy between existing and required training, and that the first require- 
ment in this process is an adequate job analysis. If no job analysis has previously been 
accomplished, the model specifies that the ISD process should be Undertaken from the 
beginning. However, if a training program already exists that is based on an adequate 
analysis, and if tasks had been selected and Job Performance Measures constructed in 
accordance with ITRO procedures, the model does not specify further how to identify the 
discrepancy. , , 

Marine Corps Order P1510.23B. The Marine Corps model begins with job analysis, and 
does not explicitly, identify a need/discrepancy analysis as part of ISD. 

Air Force Pamphlet 50-58 . The AFP 50-58 model does not contain procedures for 
a need or discrepancy analysis, but does refer to the Air Force Manual 50-2 ISD model. 
In that model, "Analyze System Requirements" is the first of five steps. This includes 
an analysis focusing on the huriian role in the total operational system, eventually reaching 
the level of specific job requirements. The ISD procedures outlined in AFP 50-58, then, 
would presumedly be initiated when such 1 a prior analysis indicated a need to develop or 
revise training. ' 
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Commentary. None of the models provides specific guidance for identifying dis- 
crepancies, other than Between the rieecl for training and the lack of a training' program. 
Although the ITRO model states that procedures will be given to Assist the dewloper in 
"determining the exact location of the discrepancy in the training program/' tro only r 
procedure actually given for existing training programs is to undertake I0D, and the only 
point recommended for entering the ISD process is at the beginning. N£ provision is • 
made for any alternative means of identifying and resolving performance discrepancies. 

Identification of Job Requirements * / K 

No decisions about training can be made until an accurate picture of the requirements 
of the job is obtained. Job analysis and task analysis, therefore, are first steps in develop- 
ing training. Job analysis determines what tasks-air performed on the job and provides 
information that can be used to establish instructional priorities. Task analysis investi- 
gates the conditions, operations, and standards that define job performance. Because job 
analysis and task analysis involve different activities, they are ordinarily performed as 
separate steps of ISD. Task analysis generally requires a lengthy and detailed specification 
of the task's behavioral elements. It is probably not efficient to undertake it before elimi- 
nating some tasks in the step, Selection of Tasks for Training. 1 

Job analysis consists of three parts: compiling a provisional list of tasks believed to 
comprise the job, verifying the accuracy of the list and adjusting it as necessary, and 
gathering task priority information. Task lists are developed in two phases to increase 
their reliability. Since ISD is a process for developing training, it .is reasonable to expect 
that trainers will be responsible for the analysis, but their familiarity with field require- 
ments can vary widely. A verification phase permits the provisional task inventory to be 
administered to a larger sample of operational personnel for validation. , 

The initial task list can be generated- in various ways, ranging from recall by a single 
subject matter expert tp extensive observation of job' incumbents. The accuracy of the 
final job description is likely to be influenced by the comprehensiveness of the initial 
inventory. Though the verification phase allows tasks to be added to the list, there is 
no guarantee that respondents' will do so. . 

.Verification may also take different forms ranging from the use of a second group of 
subject matter experts who are convened to review the original task list, to a phone or 
much broader mail survey. In concept, verification should provide for valftttion with a 
representative sample of job incumbents. Depending on the job, this usuaJg$ implies a 
larger and more varied sample, than is provided in a second group of subject matter 
experts. . * • * 

Task priority information is data about such factors as billets where the task is 
performed, percent of persons performing, frequency of performance, difficulty in learn- 
ing and performing, probability of deficient performance, consequences of deficient 
performance, and average time between training and performance. These inevitably, are 
subjective estimates. To increase their reliability, this information should usually be 
gathered from a fairly large sample of job incumbents. Where task lists are being verified 
by mail or some other form of large-scale survey, task priority information may be 
gathered at the same time. In this case, however, a second survey will be needed to 
obtain information about tasks added to the list during verification. ' 



1 8m Analysis of Tasks, p. 17, for ■ diactittion of the location of this stop in the ISD process. 
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Summary of Requirements . . 

Prerequisite Conditions and Infonnation 

None • ■ 

Procedure! 

1. Construct provisional task list. 

2. Verify and revise on basis of review/survey of job incumbents. 
-3. Collect task priority information. 

Analysis of Service Models 

ITRO Mgdej, On-site interviews (which can include observation) are recommended 
for constructing the initial task list. Methods mentioned as less effective include the 
questionnaire survey, jury of experts, and group interview. For verifying the accuracy 
of the task list, the ITRO model specifies the questionnaire survey method. When a 
current, complete task list already exists, the initial datagathering phase is omitted, and 
only the verification phase occurs. ' f , . 

r For collecting task priority information, the model recommends sending a question- 
naire to incumbents and supervisors, either at the same time as task list validation, or 
in the next ISD step. \ 

Marine Corps Order P1610.23B. Several methods are listed for collecting job datsi, 
such a» interviews, observation, analyzing the content of training programs, or assump- 
tions. The manual states that each* method has advantages and disadvantages relative to 
the others, but does hot tell What these are,, nor recommend any method over another, * 
(A single exception is the statement that "detailed personal interview" is superior to 
"assumption analysis." This is presented as an example of why a record should be 
retained of what methodology was employed.) To verify the accuracy of the task list, 
the Marine Corps model specifies that it be "double checked" by a group of subject 
matter specialists, including supervisory and instructor personnel. The use of question- 
naires to collect task priority data is not discussed; the only suggested source of this 
information is Commandant of the Marine Corps task analyses. 

Air Force Pamphlet 50-58. The AFP 50-58 model recommends obtaining a verified 
task list from an Occupational Survey, if available. If not, the model recommends using 
a team of seven or eight subject matter experts to develop the list, based on such factors 
as specialty descriptions, familiarity with similar jobs, and engineering data. Although 
implying that this method is not as desirable as an "in-depth survey," AFP 50-58 noles 
that the time and resources required for the latter are seldom available to training devel- 
opers. Interview, observation, and questionnaire are also mentioned as possible data 
collection methods. To verify the task list, suggested methods include questionnaire, 
interview, observation, simulation, and assumption. For collecting task priority infor- 
mation, the model suggests the use of Occupational Survey Reports, interviews with 
subject matter experts, and questionnaires. 

Commentary . While the ITRO and AFP 50-58 models recommend surveys for 
collecting data about job tasks, both acknowledge that less reliable and less costly methods 
may have to be used. The Marine Corps model, with the exception noted, does not 
identify any one method as superior for .compiling and verifying task lists, and contains 
almost no guidance for collecting task priority information. All the models, then, allow 
much latitude in choitfe of methods. Under these. conditions, the accuracy of the task ' 
list and the reliability of task priority' data would appear to depend on the experience 
and conscientiousness of the persons collecting, and contributing information, rather than 
on the particular model folio Wed. , 
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Selection of Tasks for Training 

»••»,. ■ 

Taikt identified during job -analysis are classified on the basis of whether training 
will or will not be provided. Those selected will become the basis of the training program 
• those rejected will not be reconsidered until the external evaluation. Thia step is based 
on the premises that (a) for some tasks identified by job analysis* it is not necessary to 
provide training, and (b) resources mgy not be .sufficient to provide training for all tasks 
on which training would be desirable. ' , 

Prerequisites to the selection -process are a list of tasks that comprise the job, « 
information about the tasks for establishing their importance and need for training, and 
decision rules for determining training priorities. To promote objectivity, the information, 
and the procedures for processing it?, should be developed prior to the selection process. 
In the absence ofa parent list of tasks, selection is not selection at all, but simply a 
- decision. If task lists, priority information, and rules of application are developed con- 
. jointly with task selection; inevitably the latter 'will influence the former. It will be 
difficult to avoid establishing priority criteria consonant with one's viewpoint on what 
is important to train; that is, one easily acquires a tendency to select criteria to fit tasks 
rather tha n selecting tasks on the basis of crite ria. This i s especi ally likely because the 
choice ofxrtterii itself is largely based on judgment; information atout the relation of 
task selection criteria to total system performance is not available. At present, there « 
no way to test the adequacy of tasks selected in terms of overall system, effectiveness. . 

Some task characteristics that are, used to determine training priorities are number 
and percent of persons performing, frequency of performance, learning diffieulty, time 
spent in performance, learning decay rate, length of time after training before perform- 
ance, and criticality (probable consequences of inadequate performance). Characteristics 
such as frequency of performance- cair be determined empirically; others, such as cri- 
ticality, by consensus. • • * - ' 

After identifying which tasks are desirable to train, and which of these are most 
important,, the requirements for training resources are considered. The final. selection of 
tasks to be trained is made on the basis of training resource requirements and availability— 
that is, cost. Some factors which affect the number of tasks for which training can be 
provided are the number of graduates required and the demands training will make on 
time, facilities, equipment, fuel, and instructional personnel. If not all desirable tasks 
can be trained, less important tasks are rejected. 

A factor that greatly influences the demands made on training resources is the 
Straining setting (i.e., resident instruction, formal on-the-job training, self-taught packages, 
etc.), although to avoid possible bias in. task selection, the setting is' selected in a sepa- 
rate ISD step, If the setting is identified before the tasks are selected, thereSnay be a 
tendency to select tasks that are easy or economical to train in that setting and to avoid 
those that are difficult and/or expensive. Also there may be a tendency to select tasks 
that have been taught in the prescribed setting before. 

Although the desirability of isolating task selection from setting selection is clear, 
there is a contradiction inherent in separating the two. Tasks are ranked in importance 
on the basis of their characteristics but selected for training on the basis of -available 
resources/cost, and cost is, in turn, greatly conditioned by setting. To completely 
separate these processes') would be to defer consideration of perhaps the greatest cost 
factor until after tasks have been selected, When this selection is ultimately constrained 
by cost. To maintain, insofar as possible, tlie'ad vantages of separation, and yet avoid' 
this error, the selection of tasks and setting must be accomplished in a series of 
interactive iterations. 
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Summary of fcequireroents * 

Q « 

fterequisite Conditidhi and Information . > « 

. 1. Taskf performed in the job are listed. * * > 

2. Information is available for establishing importance of tasks and need 
for trainirfg. * * 

3. Decision rules to be applied to task information are available: % 

- Procedures * • ■ * 

• Y 

1. Apply 4 decision rules to information for bach task to determine 
trailing priorities. ^ • 

2. Select tasks for traihi^on basis of training priorities and resource 
avaUabilityV ^ 

Analysis of Service Models / 

ITRO Model. The first step of the ITRO procedure^ to define the criteria by 
whjch task importance will be measured. Percent performing, time between job entry 
and task performance, and learning difficulty are suggested as the most appropriate 
criteria for determining the absolute importance of tasks, and criticality '(consequence's 
of inadequate performance} as more applicable to combat tasks then nonconubat. 'It 
is further suggested that the number of criteria selected be generally limited to about 
four. - Beyond such suggestions, fro guidance is given for selecting criteria. 

When the criteria have, been decided upon, questionnaires are constructed, then used 
to collect the task priority information from incumbents and supervisors. ^This may be • 
done simultaneously with job analysis, in the preceding step.) The data gathered are 
then examined to identify tasks that do not warrant training. This is a rather gross 
determination, designed to weed out tasks that are obviously not important. The model 
spates that if there is doubt about a task's importance,, it should probably be retained 
at this point. / • » 

Next, available training resources are estimated, as an aid in determining about how 
many of the remaining tasks can be trained. Then the tasks in which training is needed 
the most are selected,. up to, the tentative number to be trained; another 20% or so of 
tasks next in importance are also identified, and rank-ordered, to be available for selec- 
tion when resource availability is known precisely. The model states that tasks which 
rank lower on the task selection criteria are less in need of training, but does not suggest 
methods for weighing the relative importance of the several criteria, nor for considering 
how the different criteria might be related to total system effectiveness. 

■ Marine Corps Order JP1510.23B . The, Marine Corps model provides guidance similar 
to that of the ITRO, but more briefly stated. It presents eight task selection criteria 
but does not differentiate among them as to applicability. As noted earlier, no pro- 
cedures are described for collecting task priority mfonimtion, other than obtaining a task 
analysis report, if available. 

Air Force Pafaphlet 50-58. Three criteria are specified for use in selecting tasks; 
number performing, percent performing, and criticality. The model also provides rules- 
for applying the- criteria (e.g., if themunber performing a task were 51-100, and the 
percent performing below 50%, the task would he recommended for training if its 
correct performance would increase job effectiveness, but not if its correct performance 
would merely increase efficiency). AFP 50-5$ stresses that the rules for applying the 
criteria are only guidelines to making an "initial decision" about task selection, which 
must be verified by some other means. No guidance is orovided regarding any conflict • 
between training requirements and availability of training resources. 
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Coflftnentary . All the models, then, suggest criteria by which to select tasks for 
training, but ohly one provides a formula for their &ppiication-~and that only condi- 
tionally, andWpr only three criteria/ Moreover, the Marine Corps model's- lack of guidance 
for collecting task priority information could result in the practice of exploring priorities 
simultaneously with electing tasks, thereby neutralising Nthe advantages of collecting ^ 
priority information in a separate step. 

To specify tttorq fvilly how to select *and fc weigh various criteria, information is 
clearly needed concerning how the criteria are related to Job performance. It is not 
known, for example, how total system effectiveness is changed as a result t>f selecting 
difficult tasks, in contrast to tasks with high percentages of total time spent performing; 
or vice verm. Indeed, for almost every criterion suggested a rationale cfb\ be found for V • 
using it to, reject tasks instead of selecting them: for example, it can be argued that, 
training, should be given for more frequently performed tasks because this will insure 
that the effects of training will be more widely felt. On the other hand, it is not too 
extreme to suggest that training not be given for the most frequently performed tasks 
on the premise that incumbents will have a better opportunity to learn them on the 
job. In the absence of information about how system effectiveness is related to task 
selection criteria, rational analysis alone cannot be y*ed to select the criteria. 

Analysis Of Tasks 

v V 

1 In the context of ISD, task analysis is a description of when And how within t^/* 
job 'environment. the performance of a task is required. Thus it consists of specifying 
conditions of performance along with initiating cues, behavioral elements, and standards 
of performance.' The information generated is later used in constructing Job Performance 
Measures and in developing objectives. 

The only ^prerequisite for task analysis is that tasks to be analyzed have been iden- 
tified. It therefore can be undertaken almost concurrently with, and is considered by. 
some a part o/, job analysis. Job analysis,, however, consists essentially 6f a rational 
rec9n$truction of job requirements followed by a survey of job incumbents. fQr anal- 
ysis, on the oUter hand, requires a detailed specification of the elements of task perform- 
ance. Because different activities are inx^lved in the two functions and because task 
analysis is often a lengthy and costly process, it should usually be undertaken only for 
those tasks that have previously been selected for training. 

Though it is generally more efficient to do task analysis after task selection, this 
order of events can be a source of problems. The longer that task analysis is delayed, 
the greater the possibility that it may become confounded with decisions about training 
When tasks have already been selected for training, for example, there may be a greater ) 
tendency to describe them as they will be performed in training rather than as they are 
performed on the job. It is important to emphasise that while training conditions and 
standards (established later in the ISD process) may differ considerably from those of ^^Vi 
the job, it is not the purpose of task analysis to^det^rmine how a task will be trained. <x 
Rather it is to discover under what conditions the task is actually performed, and what 
standards of performance Yre required in the Job. Also, when selecting a task for train* 
ing means that it must be analyzed and rejecting it means that no/ analysis will be 
required, there can be a tendency to select tasks that have few behavioral elements, or on 
which the standards are easy to identify, and so forth. (A parallel is the visible tendency 
within the" field of education and training to teach those things that are easiest to explain.) 



Summary of Requirements 

^ * * , ■ » . 

Prerequisite Conditions and Information 

1. Tasks have been selected for training. 
Procedures 

1. ^For each task specify for the job environment: 

— Conditions of performance. 

— Behavioral elements. 

t Standards of performance. 

Analysis of Service Models * , * 

ITRO Model. Task analysis is inclined in the step "AnalVze Job/' which specifies 
that the conditions, initiating cues, standards, and elements of each task fce listed. The 
manual provides a Job Data Worksheet that can>s-used for this purpose. The informa- 
tion is to be collected in the. same manner ahp from the same sources as data for the 1 
task inventory. This can be done at the time of the original inventory, for till tasks, or 
Jater for only those tarks selected for training. - 

Marine Corps Order P1610.23B. The model specif ies^hat task elements be identi- 
fied along with tasks when job analysis is conducted. A recommended hierarchical for- 
mat, the Job Analysis Sheet, is illustrated. However, standards, conditions, and initiating 
cues are not recorded on this Job Analysis Sheet, but are "preserved to be used during 
step 1.3, Construct Job Performance Requirements and Measures." During step 1.3, they 
are written down on a different form, the Consolidated Job Data Sheet, along with some 
other information concerning the task. 

Ak Force Pamphlet 60-68. The model specifies that tasks be divided into "aub- 
tasks" as part, of job analysis, but no other task analysis is done until tasks are selected 
for training. Subtasks are identified to create task units nearly equal in complexity and 
amount of implied activity, for ease of. analysis later. After tasks are selected, elements 
("actions"), cues, standards ("proficiency requirements"), gme required, and other aspects 
of the task, such as precautions to observe, are recorded in any one of four general far- 
mats shown in the manual. 0 * 

The AFP 60-68 model does not discuss task conditions per *e. In the illustration of 
the use of a recommended format for task description, however, two of the items are 
similar to conditions. Under the heading "Activity Support Elements" for a task involving 
the use of a calculator, are listed the calculator, operator's manual, liaft^of values, etc. 
Under the heading "Support Information" are listed such items as "Assume values are 
dollars . . ." arid "Assume prior orientation to calculator." 1 In this latter case, however, 
it appears that the description of the task as performed on the job is beginning to be 
confounded with a description of the conditions under which it will be trained. 

Commentary. The procedures suggested in the three models do not differ markedly . 
The use of any model could be expected to result in an adequate definition of the task. 



^ Construction of Job Performance Measures - 

After tasks have been selected for training, a test may be constructed for each task ' ' 
to serve as a means of keeping training faithful to job requirements, as a means of 
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evaluating training and the design process, of both. To serve either purpose, the test; or 
Job Performance Measure, should possess high predictive validity; that is, performance on 
the test should be highly related to performance of the task itself.' If the test is to 
serve as -a means for keeping training faithful to job requirements, the behaviors, condi- 
tions; and standards of „ the Job Performance Measure must resemble those of the actual 
task as closely as is practical. If the test' is* to be used to evaluate trairiing,it can con- 
ceivably vary uTfidelity as long as its predictive validity is maintained. 

To evaluate training design, Job Performance Measures are administered to persons 
in training, or to persons after they have reached the job! Although the latter method 
may give a more accurate measure, it is also more dostly. If trainees, or graduates, pass 
the Job Performance. Measure, the training is considered successful. If not, the training 
cannot be considered adequate; regardless of traineets' previous performance on within, 
course achievement tests. 

In attempting to keep training requirements faithful to job requirements, the Job 
Performance Measures serve as a connecting link between the tasks and the objectives of 
training. In later ISD steps, the tasks identified by job analysis, and selected for training, 
must be translated into training objectives and tests of those objectives. Some degrada- 
tion in fidelity can be expected at that .point : job conditions cannot always be repro- 
duced in training; standards may need to be lowered for training; perhaps only part of 
a task will be judged to require training. In this translation from job requirements to 
training requirements, there will be much opportunity for error. In the name of adjust- 
ing to training conditions or constraints, critical aspects pf task performance for which 
training was seen as necessary can virtually disappear. Job Performance Measures can 
serve as standards for preventing such extreme degradation. 

After, the measures have been constructed and validated, objectives can be derived 
directly from them, and achievement tests for the course can be derived in turn directly 
from the objectives. Since the Job Performance Measures have already been. determined to 
be good predictors of task performance, then objectives and tests which resemble the . 
Job Performance Measures closely can be expected to lead to successful task performance. 
Although the capacity of the Job Performance Measures to help achieve this end is 
subject to how well they are adhered to, generally the existence of such a model can be 
expected U> inhibit departures from job requirements. 

To serve their purposes, Job Performance Measures must be constructed before objec- 
tives or tests are developed. To provide an independent assessment of, or point of 
reference for, training, they must be independent of training. If they wire not constructed 
until after training objectives had been formulated, it would be nearly inevitable that they 
would reflect training to some degree. To whatever extent'fraining behavior, standards, ' 
and conditions are incorporated in the Job Performance Measures, they are that much 
less capable of measuring the job adequacy of the training product. 



1 In traditional test coiutruction, criterion-baaed validity haa been viewed as the most desirable kind 
of validity to demonstrate. Here criterion-related validity would be a statistical demonstration of the 
relationship between the test and the ultimate criterion, actual performance of the task on the job. 
Where, however, the test unjfter consideration is a performance test that is content valid, there is usually 
no better criterion of task performance available. In the absence of another criterion of known high 
validity, it is meaningless to pursue the issue of criterion-related validity. In such situations, another 
variety of predicUve validity— so-called concurrent validity-^can be sought, that is, the capacity of the 
test to distinguish performers who «• acknowledged to be effective or highly experienced from those 
who are seen as ineffective or who are novices. 
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% Not all models of ISD include the construction of Job Performance Measures; The 
alternative to deriving objectives and tests from Job Performance Measures is to derive 
them directly from descriptions of tasks, and this is what some models prescribe. It is 
difficult to foresee exactly how training development is affected by doing without a 
model or means of evaluation. Certainly the relation between training and job perform- 
ance may, in some cases at least, be so obvious that the construction of a Job Perform- 
ance Measure would seem superfluous. Considerable effort is involved in the construction 
of these measures, which may not be justified. Nevertheless, a ready means for deter- ' 
mining whether training is meeting its goals would appear to be virtually indispensable 
to a systematic approach to training design and development. 

Summary of Requirements 

Prerequisite Conditions 

( 1. Tasks selected for training are listed. 

2. Training requirements for these tasks have not been identified. 

Procedures y 

1. Construct a test for measuring the performance of each task selected 
for training. 

2. Validate each Job Performance Measure to insure .that it predicts task 
performance. 

/ Analysis of Service Models 

ITRO Model. A Job Performance Measure- is to be constructed for each task selected 
for training. The model discusses predictive validity, and states that it cannot be determined 
if there is no way to test the task under actual job conditions. In such cases, the model 
states, the physical fidelity of the Job Performance Measure will be used as the criterion 
of its adequacy. The process of measuring the physical fidelity of tb* Job Performance 
Measure is termed "verification." However, although the model<distinguishes between 
validation and verification at some points, it confounds them at others, and it does not 
adequately explain how validation or verification is to be accomplished. For example, it 
does not make explicit that, in the validation process, task performance is used as the 
validation criterion, while, in the verification process, the physical fidelity of the test 
itself is judged^no* task performance. 

Mari ne fbfps Order P1510.23B. The Marine Corps model states that Job Perform- 
ance Measures must be constructed and validated. It specifies that the validity of the 
tests should be higher for tasks in which inadequate performance has more serious con- 
sequences. 

Air Force Pamphlet 50-58. This step is not included in ISD, nor is any equivalent 
step. In a later step, objectives will be derived directly from tasks .and task elements. 

Commentary . Despite some lack of clarity, the ITRO model's explanation of how 
to construct and validate Job Performance Measures would appear sufficient to enable a ' 
test designer to produce them. The Marine Corps model, which severely abbreviates the 
ITRO explanation, and combines Job Performance Measure construction with certain 
task analysis activities ("Construct Performance Requirements ") in a single step, would 
not appear altogether adequate. Perhaps significantly, the Marine „Corps model does 
not specify later that objectives be derived from the Job Pe.formance Measures, but 
rather from the Job Performance Requirements, which are essentially records of task 
analysis. The role df the Job Performance Measure in the Marine Corps model is thus 
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somewhat unclear., As noted, the AFP 50-58 model objectives are also derived directly 
from task analysis information. In this respect, the AFP 50-58 model is consistent when 
it omits the construction of Jot) Performance Measures. At the same time, this omits 
from the model a significant mcfens of measuring the validity of instructional programs. 

/ 

Selection of Setting 

A tenet of ISD is that training should provided in the most cost-effective setting. 
Every task previously selected for training must now be assigned to a particular location • 
and .situation in which training will occur. Training may be conducted in one or in a 
combination of situations: at resident schools and training centers, at installation support 
schools, and through a variety of formal on-the-job procedures. The latter category 
includes the use of such techniques as self -teaching lessons and job aids (proceduralized 
manuals and performance guides). 

An obvious prerequisite to carrying out this step is the freedom to assign tasks to 
any of several settings. Without a range of Options, there can be no true selection. When 
only one setting is available, /there is the danger of having to assign an excessive number 
of tasks inappropriate to the setting or to forego training for ffhose tasks altogether. It 
follows that'a variety of settings must be available and that, within .each military service, 
the selection of setting(s) must be made at an organizational level which has the authority 
•to implement its decisioh. # ' 

Another prerequisite is the availability of information about the costs of training in , 
the alternative settings. ^Costs to be examined include not only such standard considera- 
tions as equipment, personnel, facilities, and supplies, but also the indirect consequences 
related to particular training strategies-for example, the effects of different experience 
mixes (trained and untrained men) in the force structure when training is conducted in 
operational units, the cost in productivity to experienced incumbents serving as instructors 
in on-the-job training, and the cost of different settings as a function of trainee charac- 
teristics (e.g., level of maturity and capacity for self -study). .Thus, one study found an 
additional year, of education to be associated with about a 10% induction in estimated 
OJT costs, and an additional 10 points of measured mental ability, with about a 6% 
reduction. 1 x 

It is clear- that this step involves an interface with other components of the opera- 
tional system, beyond the training subsystem. The impact of the size of the partially 
trained "component in the force structure and the impact on the operational unit of the 
amount of time and resources devoted to training cannot simply be ignored. A systems 
approach implies that no part of the system is free of the influence of the others. To 
assign tasks to different settings in such a way as to maximize total system effectiveness, 
then, requires knowledge of how alternative assignment patterns will affect the total 
system. ^ * 

Summary of Requirements 

Prerequisite Conditions and Information 

1. Tasks din be assigned to ahy of several settings. 

2. Information is available on costs of training in different settings. 

3. ' Information is available on effects of training in different settings on 

total system effectiveness. 

1 Gay , Robert M., Estimating the Coat of On-the-job Training in Military Occupations: A Meth- 
odology and Pilot Study, Rsnd> R1351-ARPA, April 1974. 
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Procedures 

1. Assign each talk or group of tasks to its appropriate training setting. 
Analysts of Service Models 

ITRO Model The ITRO model discusses five settings to which tasks may be assigned 
job performance aid, self-teaching exportable package, formal on-the-job tracing sZTrt 

2l?n tr?^? 1** md nM9nt Th« development of JoV^mScf 

aids in heu of training U recommended whenever task delay tolerance'and environmental 4 

^^ n8 ^° W K UM ° f when the doe * not r^uire an aidTf outeZ 

s^U 1" V WC ^?. Perforn^nce is not dependent on a high degree of physical 

skill. Otherwise, each of the other four settings is considered, in the order listed It 

w*. * C ? 0ic **. be made by firit COM «ering non-cost factors, then con- 

sultinglocal managers to revise selections as necessary in light of costs. 
nn J** ITRO model acknowledges that ordinarily not enough information is available 
IZ lf W ? * f" ow » * formulated to pmcribe specific settings. 

fT ^^.^ ,,8hould prove he lP ful m makin « logicaldecis^ons.^ 
STln. ^IH?° n ""T* ™ amount of «>P«vWon required for a task, - umber and 
™. Ik^T"' learnin « dec «y "»te, and training resources availability. The model 
EZ? 71%** the * Jt Pf n 1 ,e ° f staining a program of inrtruction is not the only cost 

Services differ in the extent to which on-the-job training systems are alreidy in effect 
they might rightly differ on selection of setting for the same task) ' 

vtti^^Tu^L*}* 10 ^ 3 *' *** Marine model indicates the same five 
22*wiJ? I model and specifies that ihey be considered in the same order, 
♦u " t „ ms ^ Uctlon " regarded ** the most expensive setting, to be used only when no 

this treatment it must be kept in mind that AFP 60-68 is intended primarily for th? 

i^T 8 "2 technical trainm « Personnel responsible for developing resident' 
"mSln.i f J or ^ ^ ui a^0^ L hich describe. ISD in broader terms, Sates^it 
r,^ ™i° , * y *? m de ^«^i^valuate the various alternative, for acquiring quali- 
fied personnel. Among the alternatives to be considered are the relative suitability of 
^ W J HS T mtl aw f ment ' OJT - use of existing resident or field training course, 
and the development of new resident or field training counes. GeneraJlyT^equ^SeTt' 

-ti^hvT^" ° f rST 1 10 be qu ^ ied OVCT « lon « ^od of ttme canTTbT 
satisfied by the use of resident courses." 3 ' 

Commentary. Neither of the .models (ITRO and Marine Corps) that suggest pro- 
if ^ i °" 0f # r"! ng mcludw * ^thodology for detemin^whldf ttUng is 
wKL ^^effectiveness are to be measured and compared, the training Signer • 
will need to provide the procedures. The essence of the present procedures then is that 
formal training rfiould be avoided if a job performance aid wiU.uffic^ 
other than resident training should be considered. sewings 



l tPISD, Block 1.5, pu«. 2-4. 
J AFM 60-2, (Mr*. 3-4, 
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Development of Training Objectives and 
Objective* Hierarchies 



* 

The preceding itept, generally referred to at "front-end" analysis, have focused pri- 
marily on job requirements. The development of training objective! represents a shift 
to focus and is the first step in designing training to meet these requimments. It is a 
pivotal step in the ISD process because objectives provide the bridge between performing 
a task .and learning to perform a task. Objectives are descriptions of What a trainee will 
be able M> do following instruction. They thereby establish what behavior, when exhibited 
by trainees, will be accepted as evidence that instruction was successful. In turn, they / 
become the goals for training and the determinants of test and Instructional content. 
Training design and development from this point forward wiU.be carried out with respect 
to the objectives rather than to the job tasks themselves. 

u iff* ob J €ctivf describes »ome trainee behavior to be observed, the conditions under 
whicfr it will occur, and the standard of proficiency that will be considered satisfactory. ' 
All objectives are derived from the tasks selected for training. They describe either task ~ 
performance itself or behavior which demonstrates knowledge or basic skill required for 
task performance. 

Objectives which describe task performance are given the name terminal (or task, 
primary, or criterion) objective. In managing instruction, tasks are sometimes divided 
into subtasks. Objectives which correspond to the performance of these subtasks are 
named intermediate (or sub-, enabling, supporting, or secondary) objectives. Objectives' 
that refer to knowledge or basic skills that mediate the performance of tasks or subtasks 
are also termed intermediate objectives. 

The process of developing objectives from tasks selected for training involves several 
steps: 

• Deciding how closely capa bilities for task performance.'deflned injerms 
of behaviors, conditions, a nd standards, at the conclusion of training shall 
match the requirements of the job. Those specified in a training nhj~*w 
may be identical to those identified in the original analysis of job require- 
ments, or they may differ either because of constraints in the training situa- 
tion or because it. is not efficient to attempt to bring trainees completely up 
to job standards through formal training. 

• Identifying o bjectives in a process referred to as hierarchical analysis. Here 
a task is successively broken down into its component parts and the skills 
and knowledge that are necessary to learn or perform each "part. The analysis 
is taken down to the points where it is estimated that entering trainees would 
have the performance capabilities. The product of this analysis is a specifi- 
cation or hierarchical display of dependent and coordinate relationships within' 
a task (terminal objective), among subtasks (intermediate objectives), and 
mediating skills and knowledge (intermediate objectives) for which training 
must be provided. ^ 

• Based on estimates of the abilities of entering trainees, deciding what per - 
formance cap abilities must actually be developed during training . Selecting 
a task for training has implied that the task is important enough that 
trainees must posser* some degree of proficiency in it following training. 
Because trainees already may possess the ability to perform all or part-of a 
task, as it has been defined for training, a determination is made of which 
performance capabilities will actually require instruction. 
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Based on analysts of the task gptbiHtig; to be developed during training and 
upon wynet t of abilities of entering trainees, deciding what knowledge and 
bejic skills mmt be provided to mediate teaming and performance, teaming, 
or performing a taak may require the prior acquiaition of certain mote funds, 
mental akltta (e.g., literacy requirements mathematical skills, basic electronics 
knowledge) not observable or identifiable during taak analyaU. Like taak 
performance capabilities, these skills may exist in some degree in the 
entering trainee population. 



Summary of Requirements • . ' •• 

Prerequisite Conditions and Information ^ 

1. Tasks selected for training are listed. 

2. Information is available about training constraints that make it necessary 
to modify task requirements. 

3. Information is available about how modification of task requirements 
\ will affect training efficiency. 

4. Estimates of capabilities of entering trainees for learning and performing 
each task* are available. 

Procedures 

1. Specify taak require** enta (behaviors, conditions, standards) for training. 

2. Perform hierarchical analysis of tasks to identify intermediate training 
objectives. 

Anatysis.of Service Models 

Itfeo Model. The ITRO model specifies that every task selected for training be used 
as the basis for an objective. This is accomplished by directly translating each Job Per- 
formance Measure into, a terminal objective. This procedure automatically establishes the 
degree of fidelity of the training objective to the job task, since any necessary reduction 
in fidelity was incorporated into the Job Performance Measure. 

Next, intermediate objectives are derived by analyzing tho terminal objective in terms 
of what knowledge and skills would be required to attain it. In some cases, the resultant 
intermediate objectives, will be the same as the elements of the task itself. In other cases, 
knowledge and skills not apparent from the task analysis may emerge as well, in a learning 
hierarchy. In all cases, the analysis is continued only until intermediate objectives are 
derived which are estimated to be within the capabilities of the entering trainees. This 
implicit estimate cf entry behavior will later be' verified using tests derived from the 
objectives. , * , v 

Marine Corps. Order P1510.23B . The Marine Corps model specifies similar procedures 
tot deriving and verifying intermediate objectives. It also recognises that knowledge and 
skills required to attain the objectives but not evident from task analyses alone may emerge 
when objectives are analyzed. Regarding the development of terminal objectives and their 
fidelity to job tasks, however, the Marine Qorps model offers significantly less guidance 
than the ITRO. The introduction to the design phase of the Marine Corps manual states 
that "each taak or taak element selected for training" will become a terminal objective. 
It is not dear whether this means that some tasks and some elements, or all tasks and 
some elements, will be used. The section on developing objectives (paragraph 310) 
specified only that "the terminal objective must be related to a specific Job Performance 
Requirement/' and not which Job Performance Requirements will become objectives. 




No further guidance on this question is offered. Although the reader is directed to an 
appendix for further guidance, the appendix deals primarily with the clarity, specificity, 
and completeness of the objectives themselves, and does not include a procedure for 
deriving them from tasks. 1 . 

Nor does the^Marine Corps model contain r procedure for establishing standards 
and conditions for training objectives. In the absence of any guidance to the contrary, 
it might be inferred that the standards and conditions of the Job Performance Require- 
ment with which the objective is "directly related" should be used. This interpretation 
is not wholly satisfactory, however, since some change in the standards or conditions is 
Hlmost inevitable-eapecially for combat tasks. The lack of explicit directives for deriving 
objectives from tasks creates the possibility that products from different training develop- 
ers will vary widely in quality. 

Air Force Pamphlet 50-58. According to the AFP 50-58 model, objectives are 
derived directly from both tasks and elements of tasks. A determination is made for 
each task and element whether practice will be required in training to attain the standard 
set for it. Each task or element for which practice is judged necessary is considered a 
"new" skill or knowledge, and is accordingly translated into an objective (tasks into 
"task objectives," elements into "sut*objectives"). Those for which no practice is judged 
necessary are considered to be in the incoming trainee's repertoire. No objectives are 
developed for these. 

In the AFP 50-58 model, then, the initial estimate ofeptry behavior is used not 
only to establish the level of detail of intermediate objectives, but also to delete certain 
tasks selected for training.' The knowledge and skills that become training objectives in 
the AFP 50-58 model thus constitute a subclass of those in the ITRO and Marine models- 
ones which have been identified as (a) requiring a degree of proficiency unobtainable 
without practice in training, and (b) not in the incoming trainee's repertoire. 

Since any task judged to be within the entry population's capabilities is eliminated 
from further consideration for the proposed course,* this procedure amounts to a further 
selection of tasks for training-at least with respect to the deleted tasks. Thus the selec- 
tion of tasks made on the basis pf task priority information is ultimately subject to 
deletions made on the basis of estimates of entry behavior. 

A "survey test" will later be given to verify this estimate, but its use is not equiva- 
lent to the later ITRO/Marine step "Verify Entry Behavior." In the tHKQ model (and 
possibly also in the Marine Corps model), all tasks selected for training are subject to this 
method of verification. In the AFP 50-58 model, only tasks estimated to be a part of 
entry behavior are subject to verification. Since no objectives are derived from the 
deleted tasks, no tests are developed to measure them. The process will thereby identify 
only underestimates. Overestimation-the judgment that trainees are able to perform tasks 
which they really cannot— will not be revealed. 

. It is not clear whether* the AFP 50-58 procedures identify knowledge and skills 
that are required to attain terminal objectives but are not themselves elements of job 
tasks. At one point, in describing the possible uses of the optional Instructional Planning 
Worksheet, the manual refers to "common-element" objectives as though they belong in 
that category. "Common-element objectives are not derived frpm specific Job Perform- 
ance Requirements or Training Requirements recorded on the Training Data Worksheet." 1 
This would suggest the existence of some procedure for developing objectives beyond 
that of considering each task^and element listed on the Training Data Worksheet 
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A little further in the manual, however, this it contradicted. A step-by-step pro- 
cedure it given for deriving such objectives, beginning "(1) To identify common-element 
objective*, you must review all Training Data Worksheet* to determine the skills and ' 
knowledge that are Training Requirements."' There is a column on the Training Data 
Worksheet for indicating, for each task or element, whether it calls for any knowledge or 
skill. This rightly acknowledges that some elements— perhaps even some tasks— might 
require neither. The instructions for doing this, however, do not suggest that skills 
and knowledge are being identified which are not part of the explicit description of the 
task.. It appears more likely that the element or task is itself considered the skill or 
knowledge. 

There is also s Notes column on the training Data Worksheet, in which the trainer ' 
is instructed to explain his knowledge classification (or skill requirements). Vet, there is 
no further guidance to suggest that the explanation in the Notes column be converted 
into a separate objective. According to the instructions in AFP 50-58, any objective 
derived from this item will be a conversion of the action, conditions, and standard listed 
for the element or task on the Training Data Worksheet. 

For deciding hoV closely the training objectives shall match job tasks, the AFP 50-58 
model prescribes' a two-part procedure. The conditions and standards originally deter- 
mined for the task or element ate to be used for the objective (with necessary allowances' 
made for differences in condL ns between the job and training environments). A 
second criterion & applied, however, before the standard becomes final/ The standard 
for the objective should "reflect the proficiency level shown on the Training Data Worksheet. 

Proficiency levels are defined as " the amount (extent) of knowledge or thill required 
to perform a task or activity on the job." 9 Everything that follows in AFP 50-68, how- 
ever, makes it clear that^at least as far as training design is concerned-proficiency levels 
are training standards, not job standards. They are determined on the basis of the 'train- 
ing factors" (task priority information). They indicate how proficient the graduate is 
expected to be u|.on arrival on the job. (In some cases, the Air Force specifies two levels: 
one the desired degree of proficiency, the other a lesser degree to be achieved: in training 
when the desired degree cannot be reached because of training resource or other con- 
straints. The unit receiving the graduate is then responsible for further training.) Pro- 
ficiency levels range from "extremely limited" to "highly proficient" in skill, and, for 
tasks, from knowledge of "nomenclature" to "complete theory*." 

AFP 50*58 makes clear that, when this proficiency level is considered, the standard 
for the job will not necessarily become the standard for training. This is not wrong per se, 
but according to the AFP 50*58 model the job standard was used in determining which 
tasks would become training objectives in the first place. The criterion by which certain 
tasks and elements became objectives, and others did not, was whether the job p .oficiency 
requirement could be met by incoming trainees. For example, if the job task was to 
replace a certain component within 90 minutes, and it had been estimated that incoming 
trainees would be unable to do this without practice in training, that action became the 
performance part of an objective. If this task was performed infrequently and was not 
critical, it was later probably assigned knowledge proficiency level "a" and skill proficiency 
level "1". These would indicate that only nomenclature and simple procedures would be 
taught, and that the trainee would graduate with "extremely limited" skill, being unable 
to perform the task to "minimum acceptable levels of speed or accuracy." 



'Ibid., par.. 3-2d(l). 

a Ibid., para. 2-4f. 

J Op. cit., Vol. II, para. 3- 13a. 
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• If the trainee is not going to be trained to meet job proficiency requirements, then 
the question arises as to .why his inability to meet them without practice in training is ' 
used as the basis for choosing objectives. In this example; the trainee is unable to achieve 
the task standard before training, and he is unable to reach it after trainings Yet his 
inability to achieve If before training was the reason for translating this task into a 
training objective. •'»,«■ 

The hypothetical example is not an extreme one. AFP 50-68 specifies only the fol- 
lowing conditions under which i trainee would be expected to reach "minimum acceptable 
levels of speed or accuracy" in training: when the task is critical or semi-critical and 
(a) there is a long time between training and performance, or (b) the task is hard to learn 
on the job, or (c) the task is both hard to learn on the job* and rarely performed, in which 
case it must be critical, not just semi-critical. Many tasks, if not most, fall short of ' 
these criteria. Furthermore, if a "large majority of the incoming trainees have directly 
relevant prior experience on most elements of the task or activity," the proficiency levels 
for training are to be reduced one level-thus widening the gap between job standards and 
training standards still further. 1 It is easytoeee how prior experience could enable 
trainees to meet specified standards in less time, but it is not clear why this would be a 
reason to lower the standards. 

Commentary . Of the three models under discussion, the ITRQ provides the most 
comprehensive, explicit, straightforward procedures for translating job requirements into 
training objectives. The Marine Corps model is unclear regarding how closely the objec- 
tive! must match job requirements. The AFP 50-68 model permits the omission of any 
task or tasks which the trainees are estimated to know, 6 without, testing the, estimate. Either 
of these "loopholes", as it Were, might encourage what the Marine Corps model itself • 
rightly identifies as a "weak practice"-writing learning* objectives to match what is being 
taught, rather than to meet job requriements. 



Development of Achievement Tests 

When training objectives have been specified, it becomes possible to develop objective- 
referenced achievement tests. Such, tests can be used for many purposes, such as deter- 
mining the capabilities of the entry population, determining the effectiveness of training, 
and diagnosing student performance during training. They may take a variety of forms; 
depending on the nature of the objective being measured: performance tests can be used 
to assess behavior,' paper-and-pencil tests to assess knowledge, and so forth. t 

A particular requirement of the ISD model is that tests be developed directly from 
objectives, rather than from the content of lessons'. There are several reasons for this. 
Primarily there is the need for determining whether trainees have mastered the objectives. 
This is central to the ISD process. Also, the procedure helps insure, indirectly, that the 
content of a teeson/fcill support its objective, since it must prepare the trainee to pass the 
test. If tests are Based instead on lesson content, there* is nothing to insure that lessons 
will pertain to objectives. Similarly, deriving tests from objectives is expected to inhibit 
the introduction of extraneous material into lessons, since the instructional developer's 
goals are clear. 

Summary of Requirements 

Prerequisite Conditions and Information * 

1. Training objectives have been specified. 

2. Instructional materials have not been developed. 

'Ibid., Fij. 3-14. 
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1. Determine appropriate types of test* bated on characteristics 
of objectives. 

. 2. Construct tests to assets attainment of objectives. 

Analysis of Service Models 

ITRO Model. At least one achievement test item for an end -of -training test must be 
derived, from the Job Performance Measures, for each .terminal and each major intermedi- 
ate objective. Ii is not clear whether more than one item per objective is recommended. 
Guidance is not provided on. sampling requirements fori testing the transfer, and applica- 
tion of information, Concepts and rules, and skill. The actions, cues, conditions,, and 
stand aids of these items are to be identical to 4 those of the Job Performance Measures, 
unless additional constraihts in the training situation require a reduction in fidelity. In 
addition, entry tests (to determine qualifications for entering the course), pretests (to be 
used for trainee placement within the course), and other within-course tests are developed 
if needed. ^ 

* Marine Corps Order P1510.23B. Guidance similar to the ITRO model- is presented 
in an abbreviated form. A test item is to be prepared for every terminal objective, and 
every enabling objective. • • » 

Air Force Pamphlet 50-68. The model requires first considering what resources are 
available for teat development, and, if tests cannot be developed for all objectives, selec- 
ting the most important, according te a weighting formula provided.* The model recom- 
mends developing a Course Criterion Test with items derived from task-level objectives, 
a Diagnostic Test (a pretest for placement purposes) with items derived from intermediate 

. objectives, and a Survey Test comprised of both. 'The purpose of the Survey Test is to 
verify they assumptions about student entry behavior made when objectives were developed. 

Commentary. All of the models require that tests be developed from objectives, 
rather than from the content of lessons. All provide some information about test con- 
struction, ranging from descriptions of different types of tests to discussions of such 
conventional topics as. reliability and validity. The ITRO and Marine Corps models, how- 
ever, provide little guidance concerning what an adequate test of an objective would be. 
They do not indicate how item content and form are to be selected or how content is 
to be sampled to test for the transfer and application of skill and knowledge/ All models 
lack procedures for maintaining congruence between the actions and, processes implied 
in an objective (e.g., remember information, use information, remember concepts, use 
concepts) and the 'actual requirements imposed by test items. 1 

% 

\ 

Identification of Entry Behavior ' 

As soon as tests have been developed, it is possible to verify the, estimates of trainee 
entry behavior which provided the basis for deriving objectives in an earlier step. At 
that time, tasks were analyzed to identify their component parts and the skills and knowl- 
edge necessary to perform each part. In orderTb avoid reducing objectives to trivial 



1 For a procedure to sssssi the consistency of requirements among objectives, instructional activi- 
ties, and last* sea Wulfeck, Wallaca H M II, *t al. t The Instructional Quality Inventory: I Introduction 
and Overview. Special Report 79-3, Navy Personnel Research and Development Center, 1978. 
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levels, an estimate was made of the performance capebilitiet, skills, and knowledge enter- 
ing trainees would already possess. That estimate is now verified, before instructional 
development proceeds. 

Since individuals who will be trained with this instruction probably are not available 
at this point, trainee entry behavior is determined by measuring performance of a group 
that is representative of the entry population in aptitude, experience, previous training, etc. 
If any objectives are found to be already a part of the incoming trainees' repertoires, Jthey 
can be deleted from the proposed course at this time. On the other hand, the representa- 
tive trainees may fail to attain objectives that had been estimated to be within their capa- 
bilities. These objectives are then analysed further to derive lower order objectives; 
tests are developed for these subordinate objectives, and the new estimate of entry behavior 
is verified by administering the tests to another representative sample of trainees. This 
cycle is continued untU> set of objectives is obtained that is consistent with the verified 
estimate of entaMxtiavior. It is this refined set of objectives that will be used in subse- 
quent ISD steps. 

Uter/after the instructional program has been implemented, entry tests and pretests 
can be used to measure entry-capabilities directly. If these capabilities vary from the 
final estimate made in the preset step, objectives can be added or deleted as necessary at 
that tune. A ' • 

Entry behavior is also assessed to diagnose trainee needs when instruction is individ- 
ually prescribed. Here objectives are not modified, but rather training content is adjusted 
according to individual requirements for attaining the objectives. 

Summary of Requirements 

Prerequisite Conditions and Information 

1. Objectives have been derived through hierarchical analysis of tasks. 

2. Tests are available to measure objectives. 

Procedures 

1. Identify sample that is representative of trainees. 

2. Administer tests to sample, and determine accuracy of earlier estimate 
of entry behavior. 

3. Add or delete objectives as indicated by test results, and repeat cycle. 
Analysis of Service Models 

1TRO Model. The end-of-course test is to be administered to a sample of 25 to 30 
trainees representative of those who will undergo the proposed instruction. The results 
are then used to revise the objectives, Objectives which the sample have already mastered 
are deleted. Those which were erroneously assumed to have been mastered are analysed 
to a lower order; tests are then ad min ist er ed to another representative group, and the 
cycle is repeated until the objectives match the trainee entry level. When the trainees 
differ on whether they have already mastered an objective, it is recommended that pre- 
tests be developed to identify the entering trainees who do not need instruction. If it 
appears that the prerequisites for entry into the instruction are not sufficient to guarantee 
that trainees will possess necessary skills or knowledge, an entry test may be developed 
as well. 

Marine Corps Order P1510.23B. The Marine Corps procedures are similar to those 
of the ITRO model, although test sample rise is not specified. In addition, the Marine 
model specifies that all objectives corresponding to skills that all Marines in an MOS 
must perform correctly ("critical skffls") shall be retained in the instruction, regardless of 
the performance of the trainees in the representative sample. j 
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Air Force Pamphlet 60-58. The AF? 50-68 procedures are also basically the same 
as those in the ITRO model. The recommended minimum sample sise for verifying 
entry behavior is ten. There appears to be a contradiction between the guidelines for 
using testTpsults and the manner in which the objectives were developed. It will be 
recalled-tnat tests were developed only for skills and knowledge that were estimated 
not to be in-the entry population's repertoire. Yet the model specifies that instruction / 
shoilld be increased as needed whenever the students in the representative sample have 
greater difficulty than anticipated uv exhibiting the task behaviors. If the behaviors 
were estimated to be outside their repertoires, then by definition it was anticipated the 
students would not exhibit them at all. 

Commentary. Except for the contradiction mentioned in .AFP 50*58, the procedures 
of any one of the models, if followed, would appear to be effective in adjusting the 
objectives to match trainee capabilities. 



Classification of Objectives and 
Selection of Instructional Activities 

Before instruction is written— even before any instructional medium is »elected— the 
specific kinds of activities necessary to provide for learning are identified. * In later steps, 
media will be chosen ana instructional materials developed to support these activities. 

Some activities are recognised as appropriate to all types of instruction, (e.g., inform 
the learner of the objective, elicit his .performance, provide feedback): The most effective 
instructional activities, however, also differ according to the nature of the capability to 
be acquired, such as information, cognitive skill, or motor skill. Tljus, information is 
acquired more readily if trainees are required to restate it in a context of related informa-' 
tion; visual discriminations are learned more readily when critical stimuli are presented ' 
in different surrounds; acquisition of motor skill requires practice, and so forth. 

Before instructional activities are selected, objectives jure classified according to the 
type of capability they represent. Information about appropriate activities to promote 
learning for each type of capability is also essential. When objectives or groups of 
objectives -have been classified, and the corresponding types of instructional activities 
identified, the activities themselves are specified (e.g., "Display varing views of the 
ships. . . . Provide three separate opportunities to disassemble the servo-mechanism"). 
These specifications will later be used to determine appropriate media and the content 
of instruction. 

The choice of activities and the degree of detail required in specifying them will be 
greatly influenced by the- nature of the behavioral requirements inherent in' the objectives 
and how apparent they are. Some requirements will be self-evident in the conditions and 
standards of the objective. In other instance* '* will be necessary to analyze the objec- 
tive (and perhaps be more specific in stating conditions and standards) before its behav- 
ioral requirements can be completely identified. When requirements are not immediately 
obvious, instructional activities must be specified in greater detail to provide sufficient 
guidance for selecting media and developing instructional materials in later steps. 

Summary of Requirements 

Prerequisite. Conditions and Information 

1. Information is available about types of instructional activities appropriate 
«to acquiring different types of capabilities. ~" 
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Procedures 

1. Classify Mch objective or group of objective! according to type 
of capability. 

, 2. Specify instructional activities for each objective according to type . 

of capability. ■ \ 

k 

Mimyin or ovrvics iwooets 

# 

All three model* recognise four instructional guidelines as applicable to all types of ~~ 
learning: inform the learner of the objective; provide for active practice; provide guidance 
and prompts; provide feedback quickly. The models differ in the amount of additional * 
guidance given for specifying instructional events for different kinds of learning, and also 
in the degree to which such specification is required. \ 

ITRO Model. Bach objective is to be classified as one of 11 types of learning (iden- 
tifying symbols, performing gross motor skills, learning attitudes, etc.).' Descriptions and 
examples of each learning type are provided. Next, it is recommended that about four 
learning guidelines bsT selected for each objective from among those specified, the number 
depending on type of skill (e.g„ "Make the learning sctivi tier relevant by making them 
similar, to real life tasks that the student will be performing on *he job. ... If students 
make incorrect actions or begin tq develop bad habits, present a penalty following these 
improper responses until they disappear. . . . Display distinctive features of the pattern"). 
If these pr»ve Insufficient to assist learning, more can be added later. Bach guideline is 
then converted into specific instructional activities unique to the objective (e.g., "Call 
attention to the differences between the two ships"). % ' ^ 

The' ITRO model requires recording the type of learning amHSonesponding activities' 
on the Learning Objective Analysis Worksheet (although the forma>ulustrated in the 
manual provides space only for the learning category and not forthe activities). 

Marine Corps Order P1610.23B. The Marine £orps model acknowledges that "learn- 
ing activities will vary depending upon the type of objective bettfi taught," but does not 
indicate specific types, nor any learning guidelines specific to different types. It does 
require that whatever instructional activities are specified be recorded on the Consolidated 
Training Data Worksheet. ( ^ 

Air Force Pamphlet 60-58 . The model states that "the lesson plan developed from 
the POI [Program of Instruction] should contain a complete, concise descriptions 
instructional events for each stage of learning." 1 The development of .the POI is part of 
instructional design/while the lesson plan, apparently, will be developed by another party. 
Strictly speeJdng^hen; AFP 50-68 does not require instructional events to be specified as 
part of the I8D 'process. However, a format is provided for such specification— the Instruc- 
tion Planning Worksheet-to be used at the option of the training designer. Furthermore, 
AFP 50-68 provides guidelines for helping the student in six different types of learning. 
The manual discusses the relative advantages of part-task and whole-task practice*, dis- 
tribution of practice and rest, feedback, cues, and similar instructional factors. 

Commentary. The ITRO model presents a large array of different "teaming guide- 
lines," covering 11 types of learning, from which instructional designers Ban select, while 
the AFP 60-68 model gives more prescriptive guidance and explanation for a smaller 
number of learning types. The Marine Corps model would not appear to provide suffi- 
cient guidance or direction to insure that this step is carried out. 
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1 AFP 60-68. Vol. IV, pan. 4-8c. 
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Sdtcttofi of Instructfonel Method 



After instructional activities have been specified, instructional methods or delivery 
systems ait selected. These are the wmyt in which trainees will be brought into contac^ 
with the instruction. These decttons involve such considerations at instructional group 
fixe and location (•*., large^group, seminar, individual study); degree of individuaiisation 
of curriculum (e* v , fixed curriculum, remedial loops, branching programs); pacing (group 
or individually paced); management of the course (aj., instructor-managed, computer- 
managed, self-managed, media-controlled); and who will provide one-to-one tutoring If 
required (e.g., instructor, peer, assistant instructor). Obviously, some methods are inter- 
dependent; it would be impractical to combine a high degree of Individualization with 
group pacing, for example. 

Methods are selected so as to maximise the effectiveness and efficiency of instruction, ^ 
given the particular instructional activities, the nature of the trainee population, the setting, 
and administrative requirements and constraints. The choice of methodi is influenced to 
a large degree by the kinds of instructional activities specified in the previous ISD step. 
For example, if trainees are to observe a live fire demonstration, it will be convenient - 
to bring them together in a group; if they are to practice the procedure for preparing an/ 
accident report, individual study or peer instruction might be more appropriate. The 
choice of methods, in turn, will influence the selection of media in the next ISD step. 
If self -paced, Individualized instruction is specified, for example, a Uve lecture will proba- 
bly be precluded, but a videotaped one might be acceptable. 1 

v Within the constraints of the factors just mentioned, methods are chosen on the 
basis of efficiency, individuaiisation, self -pacing, computer management, and similar methods 
have the potential for reducing the average time in the course, thus reducing total costs 
when large numbers of people must be trained. Use of such methods can raise develop- x 
ment costs, however,' and may not be efficient for training smaller numbers. The choice 
is also influenced by level .and homogeneity of aptitude and experience of the trainee 
population, the probable life of the course, and the availability of facilities and equipment . 
Another consideration is setting; if the training is to be presented on the Job, for example, 
peer instruction is often appropriate, while combuter-nianaged instruction might be 
impractical. 

C&mmary of Requirements . f 

Prerequisite Conditions and Information 

1. Setting has been specified. 

2. Trainee characteristics have been identified. 

3. Instructional activities have been specified. 

4. Information is available on how the costs and effectiveness of alternate 
methods vary for specified settings, trainee characteristics, and instruc- 
tional activities. • 

✓ 

Procedures - 

v ' 

X. Specify the methods of instruction to be employed for eaci\ objective or 
group of objectives. 

'in the different ISp models, distinction^ between tWn^onal activities, media, sad methods vsry 
considerably, end some models combine certain of thess stepe. These differences are largely definitional 
and ars to be expected, given the interrelated nature of these elements and the proceesst through which 
they art Identified. For clarity, these elements are d isc uess d here as different entitles originating in dis- 
cretely different steps, f 
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.Analysis of Sanrioe Models 

ITRO Mod*. The ITRO modtl Include selection of methods ii the step "Specify 
Instruction Management Plan and Delivery System," the same step in which media are 
chosen and a System Master Plan is developed. The model suggests that self-paced, 
individualised instruction is generally more effective than group-paced instruction. 
Different methods of managing students are presented for both group and individual 
modes. A glossary of methods and instructional devices is provided. 

Marine Corps Order P1610.23B. The Marine Corps model also includes methods 
selection along with media selection: The model provides a matrix indicating some 
advantages and disadvantages of several methods. It is recommended that alternatives to 
the lecture be used if possible. 

Air Force Pamphlet 6048 . The AFP 50-58 model contains a Methods Directory 
describing the characteristics of several methods and their applicability to different 
settings. The model nbtes that the stage of learning (early, middle, or late) will influence 
the choke of methods. 7 

Commentary. AH of the models-describe and discuss different methods, and all 
encourage the consideration of alternatives to the traditional group-paced classroom lec- 
ture. While they specify the requirements which must be met (in terms of setting, -group 
site, etc.) before a particular method can be used,, they provide little information on which 
to base decisions about the relative effectiveness of different methods, either in general or 
with respect to their use with different trainee populations. Thus', while the Models may 
be sufficient, for rejecting inappropriate methods, they do not appear sufficient for selecting 
optimal methods. 

' Selection of Media 

After instructional activities end training methods haye been specified, and before 
instructional materials are developed, it is necessary to identify wha^media will provide 
an effective and efficient means for presenting the subject matter to the trainee. For 
example, if it has been decided that trainees should understand how the; working parts of 
an 'engine operate, it must now be determined whether an operational engine, simulator, 
television or' motion pict ure, still picture, diagram, computer-generated Image, or other 
device is appropriate and, of those which art appropriate, which is most economical. 

The nature of the instructional activities already specified determines which media 
can be considered. These activities wen selected to expose the trainee to stimulus events 
and provide response opportunities that would promote learning. To be appropriate, 
media must allow those stimulus events and response opportunities to occur. If the ' 
instructional activity required the trainee to distinguish between naval signal flags, color 
would presumedly be required, but not motion If 1 the activity was to distinguish between " 
target returns and noise on a radar scope, then motion would be required but not color. 

Which medium is the most economical is a* function of several factors, the first 
being the cost of the medium itself. Under most circumstances, for example, color 
costs more than black and white, motion costs more than no motion, audiovisual media c 
east more than printed forms. The previously selected methods also affect the choice 
of medium. For example, if self-paced Instruction had been designated, tape/slide rather 
than lecture would be used to present verbal information; if group instruction v had been 
designated, lecture could be used. Among other factors contributing to cost are existing 
investment in equipment, number of personnel to be trained, and availability and location 
>f production facilities. ' - . 
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The procedure is to determine first which media satisfy the requirements imposed 
> by' the learning activities and methods' jhat have been specified, and only then to consider 
costs. One prerequisite to selecting nte^a, then, is information about the suitability of * 
different media to different instructional activities and methods. Another is information 
about media costs and administrative factors that affect costs. In general, media are 
not selected dn the basis of effectiveness per $e since known characteristics that would 
make one medium more effective for certain instructional activities or methods are likely 
to have been incorporated when the activities were specified. 

Summary of Requirements 

Prerequisite Conditions and Information . 

1. Training designers are free to select from a range of media. 

2. Instructional activities have been specified. 

3. t Methods of training have been specified. 

4. ' Information is available concerning the appropriateness of different media 

to implement different activities to be used in conjunction with different 
methods. 

5. • Information is available concerning the costs of different media. 
Procedures 

1. Determine which media will be suitable U^implement the instructional 
activities and methods. 

2. Consider relative costs of media determined above and select most economical 
set of media. ^ __. 

Analysis of Service Models 

ITRO Model. A two-part procedure is specified for selecting media. In the first part, - 
matrices are utilized to select a set of possible media, based on the stimuli inherent in the 
learning activities, the complexity of the % objective, the training setting, the development 
site, and the amount of available funds. ' In the second part, final selections are made on 
the basis of cost, practicality, existing investment in production facilities, resistance^ 
innovation, and characteristics of the trainees such as reading ability. The first part of the 
procedure, then, is algorithmic, while the second part is judgmental. 

Marine Corps Order P15i0.23B. ^The Marine Corps model states that the use of media 
can "substantially enhance the leanfing process" 1 by stimulating more of the students'* 
senses, presumedly in comparison to a lecture. This implies a slightly different definition 
of "media** from that of the ITRO model. In the ITRO model, the "traditional classroom" 
is one of several media. The Marine Corps model treats mediated instruction as an adjunct 
to, or substitute for, traditional methods. Perhaps for this reason, the Marine Corps model 
does not treat the matter of media selection in great detail. 

A selection matrix for choosing media is provided, but no directions for its use. 2 The 
matrix displays six types of learning and 13 "Characteristics/Advantages of Media," such 
as "simplicity," "large classes," "realistic," and "motivation." If the type of learning is 
given, the matrix will yield from two to 16 appropriate media. Final choices would pre- 
sumably then be made in the light of the characteristics or advantages corresponding to 
the media indicated. To use this matrix, it is fjrst necessary to classify each learning objec- 
tive according to type of learning. However, tftre is no explicit requirement in the 

1 MCO P1510.23B, para. 430.4d. 
J lbid , App. Q. 
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Marine Corp* model to do this, nor is there any space indicated on the Consolidated 
Training Data Worksheet to keep a record of type of learning. 

Air Force Pamphlet 50-68. A media selection matrix and a procedure for its uce 
are presented. The- first' step is to classify the^r fledge or skill represented by each 
objective as either concrete (e.g., motor skill, e test ifying actual objects) or abstract 
(e.g., rules uung language, verbal teaming). This is done for'early, middle, and late 
states of learning for each pbjective. 

For each .concrete skill or knowledge, the second step is to decide whether media 
~«y needed. The model states thai actual job conditions and equipment are preferable. 
i».edia should be used only w>en certain specified practical and instructional constraints 
prevent the use of job conditions. The next step u to identify which of three stimuli- 
auditory, visual, and kinesthetic— are involved in learning the skill or knowledge. A 
guide is provided to aid this determination. » The stimulus ("presentation mode") in turn 
determines the media options. When the matrix identifies more- than one medium, a 
choice is made on the basis of group size, cost, instructor's role, and other administrative 
factors listed in a Directory .of Media. 

For each abstract skill or knowledge, the second step is to determine which of four 
presentation modes— auditory, visual, semi-motion, or interpersonal simulation— is required 
to express the concept associated with the knowledge or skill. If the concept can be 
expressed by spoken or written word alone, no medium is required. Otherwise, a 
matrix is used to determine media options, based on the specified presentation mode and 
the size of the group receiving instruction. If the matrix indicates more than one medium, 
a choice is made on the basis of the administrative factors listed in the Directory of Media. 

Commentary . The appropriateness of different media to a given objective is deter- 
mined in the ITRO model by considering the stimulus and response requirements of the 
instructional activities; in the AFP 50-58 model, by considering the objectives themselves; 
and in the Marine Corps model, primarily by the type of learning. In all cases, the 
accuracy of the determination would appear to depend on the skill of the analyst. None 
of the models provides guidance on how to determine stimulus requirements— only on 
how to match them to media. 

The detailed procedures in the ITRO and AFP 50-58 models clearly represent a 
serious attempt to encourage users to consider a wide range of media. The incomplete 
treatment of this step in the Marine Corps model brings into question whether Marine 
Corps trainers are actually expected to do likewise. 



Grouping and Sequencing of Instruction 

To create a course of instruction from the diverse objectives that now exist, the 
next step is to determine' and specify how they will be grouped and sequenced. An 
obvious factor that affects grouping is commonality of subject matter. Another is trans- 
fer of learning; to the extent that seems reasonable, objectives should be grouped together 
when some transfer of learning between them can be anticipated. 

Two kinds of factors affect sequencing. One is the degree of dependency between 
objectives. If learning one objective depends on learning another, they must be sequenced 
accordingly. Where no dependency exists, more latitude is available and the process 
invokes the second factor, the set of overall sequencing principles chosen; simple to 
complex, job order, familiar to unfamiliar, most difficult first, most difficult last, etc. 
Unfortunately, there are no universal principles. In fact, a rationale can often be found 
for conflicting strategies. For example, sequencing topics on the basis of the order of 
job performance may often be most meaningful from the point of view of the learners, 



35 



yet it may sometimes result in giving the most exposure and rehearsal to the easiest and 
least important aspects of job performance. At present, decisions about sequencing of 
instruction must be based in part on individual 'judgment. % 

In addition to f a ctors that pertain-to learning, grouping and sequencing decisions 
are also subject to practical and administrative factors such as safety and availability of 
equipment apd faculties. A principal characteristic of the ISD approach is that decisions* 
about sequencing and grouping* are made as much as possible on the basis of learning 
factors, rather than solely on the basis of administrative factors or past practice. 

Summary of Requirements 

Prerequisite Conditions and Information 

1. Knowledge of the effects on learning of different sequencing plans 
is available. 

Procedures 

1. Identify commonality of subject matter and anticipated transfer of 
• learning between objectives. 

2. Identify degree of dependency between objectives. 

3. Select overall'sequencing principle^). 

4. Group and sequence objectives. 

Analysis of Service Models 

1TRO Model . Objectives are first categorized as dependent, independent, of sup- 
portive (those between which some transfer of learning occurs). The dependent objectives 
are then sequenced as necessary to insure that prerequisite learning will occur first. Next, 
supporting objectives are placed as close to each other as possible without interfering with 
the dependent sequencing. Independent objectives are sequenced last, in any order that 
is practical. The next step is to identify identical objectives and objectives with identipal 
actions. Deletions are made to identical objectives so that each is taught only once. 
Objectives with identical actions, but different objects of the action, are grouped together. 

Marine Corps Order P1510.23B . The Marine Corps procedures are identical to the 
ITRO, except that no mention is made of identical or identical-action objectives. 

Air Force Pamphlet 50-68 . The terminology used is slightly different from the 
ITRO/Marine models, but provides essentially the same procedures. 

Commentary . The only unambiguous guidance from the Service models is to 
place dependent objectives later in the sequence of instruction than the objectives on 
which- they depend. Given the inadequacy of knowledge about how different sequencing 
strategies affect learning, and the difficulty of estimating transfer of learning, this may 
well be all that the models can do, leaving the rest to the individual judgment of the 
training developer. 

Development of Plan for 
Authoring and Managing Instruction 

A written plan must be made available to those who will author, conduct, and 
manage the instructional program. Whether in a single master plan or in separate docu- 
ments intended for the different personnel concerned, the decisions and specifications 
made in previous ISD steps must be transmitted to those *ho will carry them out. 
Thus, the content and structure planned for each lesson must be specified to those who 
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will author instruction. Information about the presumed literacy level, experience, -age, 
etc.. of the target audience will need to be provided, and the type and frequency of 
testing specified. The tame informstton mutt be made available to the inetructora and 
coune managers. In addition, they mutt be informed about such administrative matters 4 
. as class schedules, Instructor allocations, and equipment and facilities use. All of this 
information is provided in a plan(s) for developers and managers of instruction. 

Although the provision of a plan is discussed here as a distinct ISD step, the content 
of the plan so far as lesson specifications- is concerned has been almost wholly determined 
by the' decisions made in earlier steps which specifies the grouping and sequencing of 
objectives, instructional activities, methods, and media If the earlier decisions and 
specifications about instructional design have been recorded in a form that successfully 
conveys this information to those who will use it, then a good portion of lesson specifi- . 
cation has been accomplished. 

What remains is to organise the course into its units/lessons, specify the objective^) 
for each unit, and describe the instructional plan for each objective (e.g., how the trainee 
is to be informed about the objective, how stimulus material is to be presented and per- 
formance elicited, how guidance and feedback are to be provided). If , however, the 
decisions made earlier have not been recorded in usuable form, this must be done at 
this point as part of the lesson specification process, prior to authoring instruction. 

In either case, decisions must be made about administrative matters such as class 
schedules and equipment use. Although such information may not be necessary to develop 
instruction, its development is included in this step for the sake of completeness. Actually, 
aspects of the plan that have no bearing on lesson specification can be developed either 
now or later, so long as the plan is complete before instruction is implemented. 

Summary of Requirements^ 

Prerequisite Conditions and Information 

1. Objectives, instructional activities, methods, and media have all 
been specified. 

Procedures » 

1. Specify the content and design of each lesson. J 

2. Specify how the instruction will be conducted and managed. 

Analysis of Service Models 

1TRO Model. The ITRO model specifies that a System Master Plan be developed 
to indicate how the course will be conducted, how students will be managed and tested, 
how instruction will be developed and evaluated, and other elements Media specifics- ' 
tions are made separately. 

Marine Corps Order P1610.23B . The Marine Corps model specifies that a Consoli- 
dated Training Data Worksheet be constructed for each objective, specifying instructional 
strategies (methods), instructional activities, logistical considerations, and the main and 
supporting ideas for the instruction. 

Air Force Pa mphlet 60-58. The AFP 50-68 model specifies that the instructional 
activities, methods, guidance to the instructor, estimated instructional hours, equipment, 
and media required be specified for all objectives in a Program of Instruction (POI). 

Commentary. Each of the models, then, requires that some plan or outline be 
provided. As mentioned, this step may be combined with others— a» it is, for example, 
in the ITRO and Marine models. The adequacy of different models is discussed in the 
four preceding ISD steps where the information that constitutes the tesson 'specification 
is developed. 



Reviefr md S e cti o n of ExittJiq Materials 

After instructional activities and media haw bam specified, and before development 
of instruction begin*, any available existing instructional materials are examined to see 1 
if they match the specifications in whole or part. If they do, such materials can be J 
utilised, with resultant savings in the next ISD step of authoring instruction. However, 
in addition to satisfying the requirements for instructional activities, methods, and media, 
such materials must also be appropriate to the expected trainee characteristic! (experience, 
reeding ability, education, etc.) and in harmony with the instructional management plan 
of the proposed course. 

The distinguishing characteristic of the ISD approach in this step is that the suita- 
bility of existing materials is judged according to the specifications that have now been 
established for instructional activities, methods, and media. Although these specifications 
have some latitude that would allow changes in order to match existing materials, this 
is only permissible if the alternate .activities, methods, and media are also appropriate to 
the objectives. 

If existing materials are to be reviewed for their appropriateness to the newly 
established specifications, it follows that this review cannot precede such specification. 
Indeed, if existing materials are examined btfore activities, methods, and media are speci- 
fied, it is all too easy for the former to influence4he latter. The ISD approach is to 
determine what form the instruction should take according to what must be learned, not 
according to what materials happen to be available. 

Summery of Requirements 

X > r 

^ Prerequisite Conditions and Information 

1. Instructional activities have been specified. 

2. Methods of training have been specified. 

3. Media have been specified. 

Procedures 

1. Examine existing instructional materials to determine whether any meet 
the specifications for instructional activities, methods, and media. 

2. Select materials or parts of materials that meet the specifications, or 
which could be efficiently revised. * 

Analysis of Service Models * 

ITRO Model. A^ter instructional activities have been identified, existing materials 
are reviewed to determine whether some of the effect of developing jnew instructional 
materials can be avoided. This review is to extend |o materials produced for other courses, 
in other Services, and in the civilian educational and training communities. Whether 
existing materials can be used in the course under development is determined by their ' 
appropriateness to the specified objectives, trai- e characteristics, learning guidelines, 1 
methods, and media. Materials that are nearly, but not entirely, adequate, may be 
revised. Moreover, some changes may be made in the specifications. The model states 
that the specifications for learning guidelines, methods, and media maybe considered 
more flexible than those of trainee characteristics or the objectives themselves. Any 
materials selected must be subjected to validation in a later step. 



1 Learning guideline* are used in specifying instructional activities, see p. 31. 
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Marine Corps Ordtr P1510,23B . TW Marine Corps model placet this step prior to 
determining methods aim madia. The adequacy of existing material! Is determined by 
their appropriatenees only to objectives, trainee characteristics, instructional activities, 
and test items. (The manual, perhaps by oversight, includes "instructional strategy" as a 
criterion. Instru c tional s t rategy , le„ methods and media, is determined in the next step.) 
Only Marine Corps materials are considered in the review. There is no mention in this 
stop of subsequent validation of selected existing materials, nor in the step ''Validate 
Instruction," where the only instructional input listed is newly developed materials. The 
placing of this step ahead of determining instructional strategy would save the effort 
involved in selecting media and methods in those cases where existing materials, using 
- existing media and methods, were judged adequate. However, a tendency might exist 
to adopt materials based on instructional strategies! that were actually inappropriate, if 
the strategies had not been specified beforehand in an independent step. 

Air' Force Pamphlet 50-58. This model does not discuss the review and adoption of 
.existing materials as a distinct fSD stop. This would not, presumally, prohibit the use 
of existing materials in a course. 

Commentary . Both the ITRO and Marine Corps models make clear that the selec- 
tion of existing materials is to be based on specified characteristics of the materials, 
rather than simply on personal judgment. In considering the absence of this step in 
the AFP 50-58 model, it should be noted that there is also an absence, in the next ISD 
step, of any specific guidance for authoring instructional materials. Since the selection 
of existing materials is in lieu of developing new ones, the model is consistent when it 
, omits both steps. <• 



Authoring of Instruction 

After existing materials have beejueviewed, and suitable ones adopted or revised, 
new instruction is developed as needed. The content of instruction is determined by the 
earlier decisions on grouping and sequencing of objectives, instructional activities, methods, 
and media and by the specification of content for individual units/lessons provided in the 
authoring plan. The actual authoring process consists of such activities as writing scripts, 
and preparing tapeVslide presentations. 

ISD models are not intended to provide specific guidance with regard to the many 
technical skills needed in authoring instruction. Rather, the step is included in the 
models to specify: its relationship to earlier and later steps and to indicate the empirical 
trial-and-enor orientation of the authoring process. 

As first drafts of instructional materials are produced, small portions are tried out 
on individuals who are representative of the entry population. The materials are then 
revised to correct weaknesses, omissions, and ambiguities. 

A distinguishing characteristic of the ISD approach to authoring materials is initially 
to include only the bare minimum of instruction, and subsequently to augment instruc- 
tion as needed. This approach is used to prevent the inclusion of extraneous material, 
which could easily be incorporated if an attempt were made to be comprehensive. 
Since instruction is deliberately designed to be-just adequate, the tryout process is an 
essential part of ISD. Initial drafts are not expected to be completely satisfactory and 
it should be found necessary to augment instruction. Only through this progression' 
can the economies of minimal instruction be assured. In a later step, quantitative infor- 
mation about the effectiveness of instruction will' be obtained using larger groups of 
trainees, and further opportunity for revision will be available. 
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Summary of ftequiftmtntt 

Prerequisite Conditiont and Information 

1. Objectives hava been grouped and taquencad. 

'• 2. Instructional activities hava been specified. 

3. Methods of training have baan specified. 

4. Media have been specified. 

5. Lesson structure and content have been planned. 

Procedures 

1. Develop lean instruction. * 

2. Try out instruction on small number of persons representative of students 

3. Revise and augment instruction as necessary. 

Analysis of Service Models 

ITRO Model. The ITRO model specifies that only a minimum of instruction be 
included in first draft materials. This instruction is then tried out on one student at a 
tome, and revisions made as needed. The ITRO manual provides general guidance for . 
writing or developing many different types of materials: audiovisual scripts, slide/tape 
programs, television, programmed texts, and other printed material*, platform lectures, 
seir-teachmg exportable packages, job performance aids, and so forth. 
4U Corps Order P1610.23B. The Marine Corps model also specifies that only 

the minimum instruction necessary to achieve the objective be used. In addition, the 
model requires that a plan of the proposed instruction ("Concept Sheet") be approved 
prior to beginning work. No specific guidance on authoring materials themselves is 
presented. The model specifies that, if possible, instructional materials be tried out on 
"a single representative learner." 

Air Force Pamphlet 60-68. The AFP 50-58 model specifies that materials be tried 
out on a. small number of students, but not until they have first been subjected to an 
internal review by other subject matter experts. The model provides guidance on 
revising materials during the tryout phase, including how to remedy different types of 
failure to learn (retention, transfer, acquisition), • * 

Commentary. All of the models, then, provide for a tryout of materials during devel- 
opment, and the ITRO and Marine models emphasise that instruction should be lean 
The absence of specific authoring guidance in the Marine Corps and AFP 60-68 models 
should not be considered a deficiency, since description of the many different technical 
authoring procedures is not intended to be included in ISD manuals. 1 ' 

» 

Validation of Instruction 

After instruction has been developed but before it is putlnto use, it must be tried 
out to see if it works-that is, to see whether trainee* attain the objectives. This is <*one 
by administering the. course (or major parts of the course), under conditions that closely 
approximate its intended use, to groups of trainees representative of the entry population 
Objective-referenced achievement tests developed prior to and independent of instructional 
development per se are the primary validation criteria. Other measures of instructional 
adequacy, such as time to complete lessons and acceptability to trainees, may also be 
obtained at this time. Instruction found to be deficient on any of the chosen criteria is 
revised and subjected to the validation process again. 
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/ ~ Summery of Requirements 

Prerequisite Conditions and Information 

1. Objective-referenced achievement teats are available. 
Procedures 

» i i .i ■ ■ 4 " 

N *• Specify achievement teat validation criteria (number and percent of 

persona in validation sample required to pass tests). 

2. Specify additional validation criteria. 

3. Present instruction, administer achievement teats, analyze results, revise 
instruction, and repeat cycle until validation criteria are met. 

Analysis of Service Models N 

il&Q Mode| f The ITRO procedure is to conduct both individual and group 
validation trials. It must first be decided what types of information will be needed to 
measure the adequacy of the materials (e.g., achievement testa, student attitudes, time 
to complete lesson) and what criterion values will be considered adequate. Instruments 
for collecting and recording the data must then be developed. 

Before trials are conducted, entry tests anil pretests, if available, are, used to select 
suitable subjects. In individual trials, while the instruction is being presented, trainee 
questions and any help given are recorded. When the trainee has completed the lesson, 
the pott-test is administered, foUowed by attitude and/or other chosen measures. Fol- 
lowing individual trials with from three yt> six trainees, the materials are revised in the* 
light of test results, student comments, and" whatever other data were collected. The model 
acknowledges that a good deal of judgment is required in diagnosing of deficiencies in 
the materials. When felt that the lessons are adequate, group trials are undertaken. 

The model provides a method for determining the sample size needed to obtain 
statistically significant results in group trials, as a function of different standards for 
mastering objectives. Possible criteria to be used in setting such standards are similar to 
those used earlier in task selection (e.g., task delay tolerance, availability of trained incum- 
bents, learning difficulty). The model cautions against being too rigid in setting standards 
however, and acknowledges that a great deal of error is present in any attempt to measure 
performance. If the trainees fajl to reach the criterion specified, the "revision cycle" 
(presumably the validation procedure) must be repeated. The ITRO model acknowledges 
that it may not be possible to complete validation before implementing the course, and 
that continual revision with the actual trainees'may be required. 

Marine Corps Order P1510.23B. The Marine Corps model specifies that instructional 
materials be tried with progressively larger groups and revised "until they are satisfactory " 
First, the materials are tested on a single representative learner, revised, and tried out V 
again until "consistent errors are removed." Second, group trials are conducted until 
"the desired mastery criteria have been obtained." The manual states that "group, in 
this case, does not necessarily mean that the trials must be conducted with a group of a 
certain. sixe." The Marine Corps model, like the ITRO, acknowledges that it may be 
necessary to validate materials' by continual revision with actual trainees. 

Air Force Pamphlet 50-58. The AFP 50-58 model also recommends a two-part 
validation procedure. In the first part, the instruction is presented to a group of six to 
ten students, evenly divided between low, medium, and high aptitude. Information 
about error rates, and time to complete, collected during this tryout is then used as the 
basis for revision. Instruction is then presented to another group of six to ten similar 
students. The cycle pt instruction, test, and revision is continued "until it is proven that 
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the students can perform to the level specified in the criterion objectives end tests," 
or until at least 20 students have been through the instruction, whichever occurs later. 

In the second part, complete instructional sequences (ideally thWentire course of 
instruction) are presented to a group of about 30. This is called an "operational tryout," 
and is conducted using the equipment, facilities, and administrative procedures that will 
be used when the course is operational. If students fail to meet objectives, the model 
requires the designer to reassess the task analysis, objectives, tests, and instruction design 
and development. * » 

Commentary. The procedures recommended in the different models are essentially 
similar, although the Marine Corps model allows considerably more leeway in the size of 
the sample for .group trials. Any oftthe three seta of procedures, if put into practice, 
would be adequate to determine whether the traintaggtbjectives had been achieved as 



After the instruction has been implemented, its actual effectiveness must be deter- 
mined. Although validation in the preceding step indicated that persons who received the 
instruction would attain the objectives, this was only an estimate, based on an approxi- 
mation of real training. How well the instruction works during actual training with 
actual trainees will now be determined. While validation occurred at one time only, 
internal evaluation will be continuous, since the capabilities of trainees and the manner 
in which instruction is conducted may change over time* The principal measure of 
instructional effectiveness* is the trainees' performance on the objective-referenced 
achievement tests. Other measures include trainees' and instructors' opinions and atti- 
tudes concerning the instruction, time to complete lessons, and attrition rates. This 
evaluation of instructional effectiveness is sometimes termed product evaluation. 

As deficiencies in the instruction are discovered, an attempt, is made to identify 
their causes and recommend. remedies. In locating the source of a problem, it will be 
important to know what ISD steps had previously been performed, what decisions were 
made, the rationales for these decisions, and the way in which they affected training. 
It would be pointless to consider revising the choice of, particular instructional activities, 
for example, without first considering whether and how they had actually been incorporated 
into the instruction. This analysis or evaluation of how the ISD process itself was carried 
out is also part of internal evaluation, and is sometimes termed process evaluation. 

Summary of Requirements 

Prerequisite Conditions and Information 



measured by the achievement tests. 



Internal Evaluation 



1. Records of students' performance on achievement tests are available. 

2. Documentation is available of what occurred during the ISD process, 
including such elements as rationales for decisions, and departures 
from standard procedures. 




2. 
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Analysis of Service Models 

ITRO M9<fol Jhe ITRO model specifies that the effectiveness of both the instruc- 
tion and the ISD effort itself be evaluated. Instructional effectiveness is measured by 
trainees* attainment of the objectives, as represented by test scores. The methods used 
are basically the same as validation prosadures. The manual also recommends collecting, 
information such as trainee backgrourioTentry skills, time to complete lessons, trainees* 
evaluations of the media and methods, and instructors' evaluation of the content. The 
product resulting from these activities is a report recommending revisions to improve the 
course. 

The other aspect of internal evaluation— progress and process evaluation-is actually 
begun earlier. It begins with scheduling the ISD steps and determining which parts of 
each step will apply to the specific project. Then, as work proceeds, reports are made 
concerning quality, adherence to schedules, departures from plans, rationales fordeci- 
sions, and other relevant factors. At the end of .each step a report is made as to "whether 
the products of that step are good enough to serve as input to ensuing steps. 

Marine Corps Order P1510.23B. TJie Marine Corps model specifies that the follow- 
ing be evaluated, preferably by a person or group independent of course design and 
instructor personnel: (a) whether students are mastering the objectives, (b) whether 
the course was developed according to required procedures and schedules, and (c) whether 
the course is being administered as planned. A sample form Js provided for use in record- 
ing trainee -test performance. In the sample Evaluation Plan, it is recommended that when 
fewer than 80% of the trainees are mastering 80% of the objectives 'the instruction will 
become suspect." Checklists are to be used to collect data on course administration. 
Student and instructor evaluations of the instruction are to be obtained. The product 
of thia step is a set of recommendations to resolve problems and weaknesses. 

Air Force P amphlet 50-58. A quality control team, free from the influence of the 
instructional staff, is to conduct the internal evaluation. General qualifications for team 
members are listed in the manual. Part of the team's resoonsibility is to use the follow- 
ing procedure to evaluate trainees' mastery of objectives: 

1. Select a random sample of students about to graduate, administer course 
criterion tests, and report the results. Before testing, the team decides 
what will be the passing score. ' 

2. Administer the corresponding diagnostic test whenever there is poor per- 

. formance on a course criterion test, to isolate the cause of the deficiency. 

3. Scrutinize the part of the course where the deficiency seems to lie, and 
suggest ways to correct the deficiency. 

. 4. After changes have been made, readminister the tests to graduates of 
the revised course. 

The model does not specify sample sizes for the test program, but requires that, over a 
period of several graduating classes, each test be given. The quality control team is 
also responsible for examining and evaluating compliance with course control documents, 
performance of instructors, and the adequacy and appropriateness of training aids and 
equipment, supplies, and facilities. 

Commentary. All of the models require that trainees' mastery of the objectives be 
used as a criterion for determining the effectiveness of the course. Any of the three 
of procedures, if put into practice, would be adequate to measure whether the 
ning objectives had been achieved. The Marine Corps model, however, provides pri- 
marily an outline of the step rather than a detailed specification of procedures. 
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External Evaluation 



In addition to determining whether trainees are attaining course objectives (internal 
evaluation), their proficiency and the adequacy of the instructional design and develop- 
ment process must be evaluated by a standard external to the course: the performance 
of the graduate on the job. If graduates are unable to perform certain tasks when they 
reach the job, and these deficiencies are unacceptable, the, course may have to be revised. 
The purpose of external evaluation is to discover any such deficiencies, identify their 
causes (e.g., improper Job analysis, inadequate training), and recommend remedies. If, 
for example, the job has changed since the original analysis, or if the analysis was iaulty, 
the list of tasks on which the course is based may have to- be changed. If the training 
product is not adequate, instructional activities or jthe objectives themselves may require 
revision. If skills and knowledge have been forgotten by the time they are needed, the 
remedy may be to increase training, train on the job, or shorten the time between train- 
ing and performance in some other way. 

Probably the' most accurate methods of external evaluation are 'direct observation 
and testing of graduates on the job. Such approaches are costly, however, so some 
reliance, if not all, must be placed on supervisors' summary evaluations (ratings) of per- 
formance. Graduates' evaluation of their own and peers' performance may be included. 
Information also may be obtained on such factors as what tasks are performed, what 
aspects of training are perceived as insufficient, and what training is not used. 

Whatever the source and type of evaluation information, it should be obtained at 
a task level of specificity. More general evaluations are of little use in isolating the 
causes of inadequate performance. In all cases, the external evaluation must take place 
fairly soon after the graduate has reached the field (usually within a few months). Other- 
wise, it will be difficult to discriminate between skills and knowledge acquired in training 
and those acquired on the job. 

Summary of Requirements 

Prerequisite Conditions and Information 

1. Access to supervisors and job incumbents is possible soon after arrival 
of graduates on the job. 

Procedures 

1. Construct evaluation instruments (mail questionnaires, job sample tests, 
interview guides, etc). 

2. Collect evaluation information. 

3. Analyze data, identify causes of deficiencies, and specify revisions. 

Analysis of Service Models 

ITRO Model. Two questions are considered in external evaluation: Can >he graduate 
perform their job tasks, and is the job the same as it was when originally analyzed? The 
recommended method of determining whether graduates can perform their tasks is to 
administer Job Performance Measures in the field 30-90 days after graduation. Basing 
decisions on data "gathered by other means (questionnaire, interview,, etc.) is described 
as far riskier. * 

Determining whether the original job analysis is still valid is seen as a matter of 
determining the present relationship between the Job Performance Measures and actual 
job requirements-Hhat is, the predictive validity of the Job Performance Measures. Per- 
formance on the Job Performance Measures is compared with supervisors' evaluations of 
graduates' performance. 
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For interpreting the information from interview! and questionnaires, and deckling 
whether the coune.nceds revision, the model acknowledges thai no hard and fait rules 
apply. It provides guidance, however, oh such questions as how to interpret data when 
a conflict exists between supervisors' ratings and Job Performance Measure results. The 
model cautions against revising simply for the sake of revision, and recommends making 
few, if any, changes if the majority of graduates and supervisors ar* satisfied with the 
quality of the training. 

Marine Corps Order P1610.28B. The Marine Corps model states that the .purposes 
of external evaluation are to determine whether the graduates are performing as 
trained and whether the job has changed since front-end analysis. The principal method, 
is to survey graduates and supervisors, by questionnaire or interview, on how well they • 
believe the' graduates perform the job, the type and extent of training received on the 
job, the effectiveness and relevance of instruction, how graduates of the new course com- 
« pare to graduates of earlier training, and similar points. In addition, the manual states 
that "the validators will attempt to determine how well the graduates scored on the Job 
Performance Measures and which JPMs gave them the most trouble/' V 

Air Force Pa mphlet 60.58. The AFP 50-68 model specifies that external evaluation 
may be accomplished by questionnaires, observation, interview, and formal supervisor's 
evaluation during the graduates' first two months on the job. Content of the evaluation 
instruments is to be based on the Job Performance Requirements and Training Require- 
ments listed on the Training Data Worksheet. Examining the data collected from valida- 
tion and internal and external evaluation is expected to show whether the graduates satisfy 
job performance requirements. 

Commentary. The ITRO method, administering Job Performance Measures to grad- 
uates in the field, appears to be the most reliable way to measure the adequacy of the 
instructional design, but also the most costly. The ITRO model provides the most guid- 
ance for isolating the causes of performance discrepancies, revealed during external eval* 
uation. If summary evaluations are to be used, the J1TRO and AFP 60-58 model both 
specify that information be obtained at the task level of specificity, while the Marine 
Corps model <Joes not. The Marine Corps model is unique among the three in specifying 
a delay of six months, rather than two or three, before surveying graduates. A delay of 
this length might reduce the meaningfulness of the findings, since much of an incumbent's 
r skill and knowledge at that point may have been acquired on the job, not in training. 

SUMMATION * k 

The adequacy of ISD and the Service guidance in general varies according to the 
kind oMSD step in question. Three general types of steps can be distinguished. 

1. Procedural steps. Most ISD steps involve the direct application of procedures 
The means for carrying them out are generally available. . Examples are Identification of 
Job Requirements, Analysis of Tasks, Identification of Entry Behavior, Development of 
Achievement Tests, Validation of Instruction. The adequacy of ISD with respect to 
procedural steps becomes a matter of the clarity and completeness of the guidance 

Both the ITRO and AFP 50-58 models (with the exceptions noted in this chapter) 
appear adequate in this regard. The Marine Corps model is in the nature of an outline 
of what steps must be accomplished than a set of instructions for accomplishing them, 
and while it cites the ITRO model as a reference, there is every indication that it is 
intended to be sufficient by itself for carrying out ISD. The highly abbreviated form of 
the guidance makes its intent harder to discern, thereby presenting wider latitude for 
misinterpretation or misapplication. 
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2. Dgcgrtong affecting total system performance. These design steps involve deci- 
sions that interface with other components of the operational system and affect force 
effectiveness and productivity. The steps are Selection of Tasks for Training and Selection 
of Instructional Setting. Tasks selected for training establish the goals of training and 
thereby determine the output of the training subsystem*. Thus, tasks must be selected on 
the basis of their contribution to overall system performance. The setting that is selected 
for training interfaces with other components of the system (e.g., it may reduce the pro* 
ductivity of supervisors who must devote time to on-the-job instruction) and should also 
be selected on the basis of its system effects. 

To develop information about the relationship between system performance and 
criteria for selecting tasks and setting requires a criterion of system performance against 
which variations in task and setting can be tested. Currently available criteria of system 
performance such as those used to assess force read ineM (e.g., personnel and equipment 
fill vs. authorisation, deadline rate, hours on equipment, training status) aire either insen- 
sitive to variations in training or not reliable. Because the effects of task and setting 
selection cannot be satisfactorily measured, ISD in general, and &■ represented by the 
Service models in particular, does not provide an adequate methodology for making deci- 
sions that affect total system performance -Thii fact is reflected in the lack of systematic 
procedures in the Service models-lo^selecting tasks and settings. 

3. Decisions affecting training system performance . These design steps involve deci- 
sions that affect the efficiency of training. The steps are Selection of Instructional 
Activities, Selection of Instructional Methods, Selection of Media, and Grouping and 
Sequencing of Instruction. Once the goals" of training have been decided, decisions made 
in these steps determine the efficiency with which these goals are attained. The present . 
state of the art. of instructional design, however, provides only incomplete information 
for making these decisions. 

Current guidelines for selecting instructional activities, methods; media, and sequence 
are rudimentary. They provide only the starting point for a trial-and-error approach to 
maximizing efficiency; which—given the number of possible combinations of methods- 
could not be expected to succeed. Thus, while ISD does provide a framework for com- 
paring alternate strategies (if time and resources permit), it cannot be regarded as a meth- 
odology for achieving optimal training efficiency, given the current state of instructional 
technology. v 



Chaptsr 3 ' 
A MAIL SURVEY 0*j ISD APPLICATIONS 

At the initiation of (his study, staff members visited headquarters organizations in 
each Service to identify the locations of applications of Instructional System Develop- 
ment and to determine lines of authority and responsibility for ISD" implementation »• 
Information obtained suggested the possible use of ISp'in many organizations, far more 
than could be individually examined in the study. 

In order to identify the organizations that were making maximal use of ISD pro- 
cedures, an ISD Activities Questionnaire (see Appendix A) requesting descriptions of 
instructional development procedures war distributed to schools and units in each Service 
that we're engaged in training and training development. The questionnaire was sent to ' 
Army schools by Training Developments Institute, Army Training and Doctrine Command; 
to Navy fleet training centers, technical training centers and training detachments by 
Chief of Naval Education and Training; to Navy air crew training units by Headquarters, 
Naval Air Systems Command; to Marine Corps schools by He-\Jquarters, Marine Corps' 
and to Air Force MAJCOMS by the Directorate of Operatior i and Readiness, Headquarters, 
Air Force. 

Because hi some instances nuyor commands reproduced the questionnaire and ipdis- 
tributed copies to subordinate units, it is not possible to specify exactly how many ISD 
Activities Questionnaires we*e distributed. The questionnaires were returned from a 
total of 209 organizations iirthe four Services. » 

In completing the questionnaire, respondents first listed aM courses, either existing > 
or under development, for which job analysis data had been compiled. Although we did 



1 VisiU were made to': 

Arm y * 

Training Developments Institute, 

Training and Doctrine Command 
Training Support Center, 
/ Training and Doctrine Command 

" N>v y 

Chief of Navjtl Operations 
Director of Naval Education 

and Training 
Chief of Naval Education and 

Training 
Naval Air Systems Command 
Chief of Naval Technical Training 
Naval Training Equipment Center 
Training Analysis and Evaluation Group 
Navy Personnel Research and Develop- 
ment Center 



r 

* 

Marne Corps 

Headquarters, Marine Corps 

Air Force 

Directorate of Operations and Readiness 
Directorate of Personnel Programs 
Air Training Command 
4444 Operations Squadron (ISD) TAC 
Air Force Office of Scientific Research 
Technical Training Division, Human 
Resource* Laboratory 
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not define ISD in terms of any specific type or sequence of activity, we did require that 
job analysis data be available for developmental efforts entered into the questionnaire. 
This was done to eliminate from the study instances in Which ISD was considered to be 
nothing more than the application of one or more design or instructional techniques with- 
out the need for deriving training from job requirements. 1 Thus the sample was restricted 
to those efforts which treated the derivation of training from specified job requirements 
as a fundamental characteristic of ISD. 

Across all Services a total of 1,$14 training courses so defined were identified 
(Table 1). 

Table 

Training Courses With Job Analysis 



Data Available 


% Service % 


Organizations 
Responding 


Courses 


Army 


16 


357 


Navy 


65 


459 


ManneCorps 


10 


75 


Air Force 


118 


923 


Total 


209 


1814 



Of the 209 organizations responding, 205 (98%) also provided information on the . 
total number of courses they conduct, which is 5,934, Thus, job analysis data were 
reported to have been compiled and available for about 31 -percent. Of these, job analysis 
data were reported to have been compiled and available" for 1,648 (28%), 

For the courses thus listed, a check list of 20 potential ISD activities was provided 
to identify the events that had occurred in course development (Table 2), Respondents 
indicated whether each potential activity had been undertaken, whether it had been 
completed, and whether products of the activity were currently available. 

The questionnaire survey was intended primarily to provide information for selecting 
development efforts to be examined during subsequent visits.* The data it generated are 
of limited value for analytic purposes since (a) respondents differed .in their interpre- 
tation of the items in the checklist, and (b) the activities reported were later found to 
be exaggerated when checked in the organizations visited. However, the data are of 
some interest since they reflect to a certain degree the general pattern of ISD applications 
that was subsequently revealed in the field visits. 

Percentages of responses for each ISD step conducted for each Service, and for all 
Services combined, are given in Tables 3-7, The greatest number of ISD steps performed 



1 Where training for specific jobs is being developed, requirements most often take the form of a 
listing of tasks to be performed. Where instruction is designed to meet educational goals, development 
may begin with a specification of more general capabilities (skill, knowledge) that are recognized to 
support many different activities. An explicit statement of either tasks to be performed or capabilities 
to be acquired is necessary for an objective determination of instructional requirements, 
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Table 2 

Checklist of Potential Activities in 

ISO Activities Questionnaire ' 

1. Analyzed probelm/system/existing course to determine need for training development. 

2. Obtained priority information about tasks (frequency, delay tolerance, criticality, etc.) 
3 Analyzed tasks into elements, conditions, and standards. 

4. Selected tasks for training on the basis of task priority information. 

5. Clustered tasks and selected instructional setting on the basis of priority information, 
resources, and output requirements. 

» ¥ 

6. Developed terminal and intermediate learning objectives. 

7> Sequenced and clustered objectives. * 

8. Developed, tried out, and revised tests. 

9. Identified knowledge and skills of entering trainees. 

10. Identified type(s) of learning required for each objective/and specified corresponding 
learning activities. 

11. Identified media appropriate' to type of learning and learning activities. 

12. Specified plan for pacing, instructor role, group/individualized presentation, 
scheduling, entry/exit requirements, etc. 

13. Reviewed /selected appropriate existing instruction materials * 

14. Authored/produced new instruction materials.- 

15. Validated instruction materials. 

16. Conducted instruction. 

17. Analyzed student performance (for internal evaluation of course). 

4 

18. Revised training according to results of internal evaluation. 

19. l .ermined if graduates were meeting performance requirements ori the job 
(external/field evaluation). 

20. Revised training according to results of external evaluation. 

— — — ■ — ■ — \ — ■ 1 

4k- 

a This rctivity was inadvertantly omittad from tha quattionnaire tant to Army organization! 
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Tablt3 



Hnmt off Rtspomas ffor Each ISO Stop in 
367 Army Instructional Dtvtlopmam Efforts 



ISO Activities 




RESPONSE CATEGORY 



Activity Undtrtaktn 
Products Avtilablt 

Activity Undertaken 
Products Not Available 
duo to Probitms 4 

Activity Sting Undertaken 
Not Complatad 

Activity Not Un<i rtaken 

Not Raportad 



65.8 75.6 76.5 77.6 38.7 35.3 30.2 29.7 34.5 28.3 27.7 37.0 - 33.3 15.7 23.5 14.0 13.2 7.8 5.9 35.1 



3.4 5.0 1.4 1.7 5.0 3.9 2.0 0.3 1.4 . 6.2 5.9 3.1 

2.5 13.5 14.3 10.3 43.7 39.8 40.9 43.7 47.6 40.3 40.6 41.4 
26.3 3.9 4.7 7.3 9.5 17.9 23.8 25.2 15.4 24.1 24.7 17.4 

2.0 2.0 3.1 3.1 3.1 3.t 3.1 1.1 1.1 1.1 .1.1. |.i 



•item inadvertently omitted from Army Questionnaire. 



2.8 2.8 0.3 2.5 2.5 0.3 0.0 2.6 

47.9 53.5 49.6 50.1 51.0 62.8 55.7 39.4 

17.4 26.6 20:2 26.9 27.4 22.7 32.5 19.7 

1 1 1.4 6.4 6.5 5.9 ,6.4 5.9 3.1 



Tablt 4 



ftroant o* Ratponsas for Each ISO Stop in 
460 Navy Instructional Davaktpmant Efforts 



ISO ACTIVITIES 



RESPONSE CATEGORY 





Activity Undertaken 
Products AvoHoMe 

Activity Undertaken 
Products Not Available 
due »i ProW#ros 

Activity Bttfrf Undertaken 
Not Completed 

Activity Not Undertaken 

Not Reported 



62.1 59.3 61.7 66.7 59.7 66.2 63.6 30.7 42.0^*7.3 54.4 39.2 49.0 



42.0 SS.8 47.0 43J 30J 28.3 49.1 



4.8 8.3 2.6 3.7 3.5 0.9 2.6 2J 3.9 3.5 4.6 4.8 0.9 0.4 3.5 0.4 2.6 1.7 2.0 1.1 ?.9 

S.0 5.2 9.1 8.7 9.4 11.1 12.0 36.8 12.2 11.5 13.$ 26.6 21.8 29.4 40.7 25.0 27.7 30.3 2M 32.0 19.8 

a 

28.1 31.2 26.6 30.9 27.4 21.8 21.8 27.7 39.9 25.7 25.5 27.4 26.3 31.2 11.8 16.8 20.7 22.2 35.8 35.3 26.7 

0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 2.0 2.0 2.0 2.0 2.0 2.4 2.0 2.0 2.0 2.0 2.0 3.3 1.4 
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Tabte 6 

of Rasponsat for Each' ISO Stop in 
75 MariM Corps Instructional Oaw t opma n t Efforts 









1 : ~ 

ISO ACTIVITIES 




* 

RESPONSE CATEGORY 


lliiiilll 


ill* it 

f/ /i 


mi 


till! 



Activity Uftdtrtoktn 
Products Avoitablo 

Activity Undtmhtn 
Products Not Amitotic 
dut to Probrtms 

Activity Sting Undtrtaktn 
NotCompltttd 

Activity Hot Undtrtaktn 

Not Rtportod 



88.0* 52.0 52.0 48.0 48.0 85.3 80.0 80.0 76.0 88.0 77.3 77.3 84.0 82.7 68.4 8U 78.7 80.0 70.7 70.7 73.5 

5.3 6.6 5.3 6.7 8.0 2.7 1.3 0.0 5.3 2.7 4.0 0.0 4.0 0.0 1.3 0.0 0.0 U* 0.0 0.0 2.7 

4.0 26.7 30.7 32.0 30.7 9.3 0.7 13.3 U 5.3 6.7 8.0 10.7 16.0 16.0 16.0 16.0 13.3 24.0 17.3 14.9 

2.7 14.7 12.0 13.3 13.3 2.7 12.0 4.0 17.4 4.0 12.0 14.7 U 1.3 13.3 2.7 |j 5.4 5.3 12.0 8.5 

0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.7 00 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.Q o.o o.o o.o o.1 
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Table 6 

Poromt of Ratporaas for Each ISO Stap in 
923 Air For co Instructional Davttopmant Efforts 



ISD ACTIVITIES' 



RESPONSE CATEGORY 



Activity Undertaken 
Products Available 




Activity Ummrtaktn 
Products Ndt Available 
due to Problems 

Activity Bting UndtrUktn 
Not Complt ttd 

Activity Not Undtrtaktn 
Not Rtporttd 



mmmm 



JiL. 



72.8 57.6 68.6 62.9 62.0 68.2 70.1 55.6 59.9 69.1 69.6 71.7 75.6 68.5 55.1 64.7 62.9 54.9 50.7 46.6 63.4 

a _ 

18.3 16.8 10.7 13,1 18.2 10.0 8.0 7.6 14.7 5.2 6.8 3.4 4.6 3.5 10.8 6.2 4.9 5.4 4.8 4.6 8.9 

4.0 6.3 6.4 6.2 8.8 11.4 11.3 19.3 5.8 11.7 10.6 11.8 11.5 16.3 23.4 19.5 22.0 25.6 23.1 22.9 13.9 

4.9 19.3 14.3 17.8 11.0 10.4 10.6 17.3 19.5 13.9 12.9 13.0 8.2 11.5 10.4 9.2 9.8 13.7 20.8 24.6 13.7 

0.0 0.0 0.0 0.0 0.0 0.0 0.0 Q.2 0.1 0.1 0.1 0.1 0.1 0.2 0.3 0.4 0.4 0.4 0.6 1.3 0.2 



N 
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Ttbto 7 



Promt off RwpontM for Each ISD Step in 1814 
Instructional Dtvtlopment Efforts: All Services Combined 



RESPONSE CATEGORY 




ISD ACTIVITIES 



Activity Undtrtaktn 
Products Availablt 

Activity Undtrtaktn 
Products Not Availtblt 
dut to Probltms 

Activity Bting Undtrtaktn 
Not Compltttd 

♦ 

Activity Not Undtrtaktn 
Not Rtporttd 



f / 1' I fill I Hi ft* It li 

m il m i! H i u h 




69.3 60.4 67.7 63.6 56.2 62.0 61.0 46.2 51.0 58.9 574) 564) 67.7 54.1 44.6 55.0 50.0 44.9 38.1 35.0 



55.0 



11.4 11.9 6.6 8.2 11.5 6.2 5.2 4.6 9.0 4.8 6.0 3.5 3.3 

» 

4.0 8.2 9.7 8.7 16.7 16.8 17.1 28.3 15.4 17.0 17.1 21.2 14.7 

14.9 19.1 15.4 18.9 15.0 14.4 16.1 21.0 23.3 18.5 18.3 17.6 13.8 

0.4 0.4 0.6 0.6 0.6 0.6 0.6\ 0.9 0.8 Oil 0.8 04) 0.7 



1.9 7.0 3.3 3.6 3.7 3.0 2.7 5.9 

ft 

25.9 33.4 26.7 28.7 31.3 32.5 31.4 20.2 

17.2 14.1 13.0 15.7 18.2 24.3 28.3 17.9 

04) 0.9 2.0 '2.0 1.9 2.1 2.6 1.1 
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(i.e., highest mean percentage of responses for ail potential activities in the category 
"activities undertaken/products available") is reported by the Marine Corps (73%), fol- 
lowed by the Air Force (63%), the Navy (49%), aniTthe Army (35%). 

The relatively low oveAll incidence of I8D activity reported by the Army is a 
consequence of their current focus on developing exportable training and evaluation 
materials for use in operational units (e.g., Soldier's Manuals, Training Extension Course 
lessons, and Skill Qualification Tests), rather than instructional development per $e (see 
Chapter 4). This is seen more clearly when the percentages of responses for liferent 
types of ISD activities are examined. Three major phases of FSD are (a) analysis of job 
and training requirements, <b) development of instruction, and (c) evaluation of instruc- 
tion. Items from the activities checklist that most clearly represent each of these phases 
were selected. The mean response percentages l'n the category "activities undertaken/ 
products available" for each phase by Service are shown in Table 8. > 



Table 8 

Level of Activity in Three Phases of ISD 



Service „ 


Mean Ptrctnt of Responses Indicating 
Activity and Products 


Analysis' 


Development** 


■ Evaluation 0 


Army 


76.6 


30.6 


9.4 


Navy 


57.9 


60.4 


40.0 


Marine Corps 


50.7 


82.6 


72.9 ^ 


Air Force 


63.0 


65.5 


56.2 


All Services Combined 
* 


63.9 


§9.9. 


44.2 



, Based on items. Obtained priority information about tasks (frequency, delay 

tolerance, critical.ty, etc.); analyzed tasks into elements, conditions, end standards; and 

selected tasks for training on tha basis of task priority information 

b * 
Based on items: Developed terminal and intermediate learning objectives; sequenced 

and clustered objectives; identified type(s) of learning required for eech objective and 
specified corresponding learning activities; and identified media appropriate to type of learning 
and learnirg activities. . 

c Based on items: Validated instruction materials; analyzed student performance (for 
internal evaluation of course); and determined if graduates were meeting performance require- 
ments on the job (external/field evaluation). V 



In Army- applications, only 35 percent (Table 3) of the responses for all types of 
activities indicated actions and products, whereas activity is shown in 77 percent (Table 8) 
of the steps concerned with analysis of job and training requirements. Front-end analysis 
is necessary in the development of Soldiers Manuals and prescribed for the construction 
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of Skill' Qualification Testa. 1 The pattern reported by Army organizations clearly reflects 
these activities. 

The other noteworthy aspect of these data is the relatively low level of activity in 
evaluation. With the exception of the Marine Corps (with data representing a smaller 
number of applications and a far smaller number of organizations), all Services report 
a lower level of activity in evaluation than in either analysis or development. Also less 
activity is reported by the Army, Navy* and Air Force for any of the three subclasses"' 
of evaluation (validation, internal evaluation, external evaluation) than for analysis or 
development (Table 9). External evaluation was reported least frequently. 

/ 

Table 9 

Level of Activity in Three Types of ISD Evaluation 



Service 


Percent of Ret pontes Indicating 
Activity and Products 


Validation 


Internal 
Evaluation 


External % 
Evaluation 


Artr y 


15.7 


14.0 


7.8 


Navy 


42.0 


47.0 


30.9 


Marine Corps 


69.4 


78.7 


70.7 


Air Force 


55.1 


62.9 


50.7 


All Services Combined 


44.6 


50.0 

t 


38.1 



1 William C. Osborn, Roy C. Campbell, and J. Patrick Ford. Handbook for Development of Skill 
Qualification Te$t$. HumRRO Filial Report 77-1, January, 1977. 

Individual Training *rfd Evaluation Directorate, U.S. Army Training Support Center. Guidelines for 
Development of Skill Qualification Tegt$. December 1977. 
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Chapter 4 

SERVICE METHODS OF ORGANIZING FOR ISD 

The Services differ widely, both among and within themselves, in the ways in which 
they have organized Ho meet the requirements of Instructional System Development. In 
part, their responses reflect different notions of what the ISD requirements are, and in part 
simply the different situations faced by the four Services. At the most universal level, 
there is recognition of the need for information about the ISD process and for" expertise 
to carry out its individual steps. One response to this need has been to revise instructor 
training courses to concentrate on skills and knowledge related to ISD rather than on the 
traditional content, methods of effective instructional presentation. Another response 
has been to specialize, partitioning the ISD process into several phases and assigning per- 
sonnel to develop expertise and to work on only one phase. Another approach has been 
to assign personnel permanently to ISD duties, with or without specialization in a single 
phase. Expertise has also been brought to bear by engaging civilian contractors to 
accomplish ISD or to participate in joint military /contractor development efforts. 

The need for expertise, or the lack of it, is also reflected in the degree to which 
ISD activity has been centralized. One view is that expertise can be acquired by instruc- 
tors at all training sitei, and that they should be required to apply ISD as they develop 
and revise their own courses. At the other extreme is the view that expertise is both 
limited and hard to acquire. This has led to the deliberate placement of personnel at a 
limited number of sites, to develop training to be conducted at other locations. This 
permits the use of personnel who already possess backgrounds in training design or devel- 
opment, and who can be expected to acquire further expertise through a succession of 
ISD efforts. Centralizing ISD activity in this way, however, means that instructors must 
be willing to accept training material they did not develop themselves. Resistance to 
doing so is widely acknowledged, and is sometimes termed, aptly, the "not-invented-here 
syndrome." 

In addition to the need for expertise, there is some belief in the desira jility of 
performing various steps separately in order to maintain the high degree of objectivity 
implicit in ISD. To keep the determination of job requirements, for example, insulated 
from the influence either what has previously been taught or what certain persons 
believe should be taught, some training personnel believe that job task lists should be 
developed by persons other than instructors. Another example is the obvious appropri- 
ateness of assigning the responsibility for evaluating training to others than those who 
develop it. 

To separate the ISD functions requires, of course, that each party be willing to use 
the products developed by others. This situation is analogous to that of centralized 
instructional development, and the degree of resistance may be just as high between job 
analysts and training analysts, for example, as between developers of training and 
instructors. - * 

Another difficulty inherent in separating ISD functions is achieving the appropriate 
degree of communication between those performing separate steps. Too much commu- 
nication threatens *x> compromise objectivity, but some is necessary both because the 
steps are so interdependent, and because subject matter experts (often the course instruc- 
tors) must provide input at several points. 
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The manner In which the individual Services have organized to -apply ISD is sum- 
marized in the following sections. 



ARMY * 

Training is developed at the TRADOC resident training center proponent for a 
particular MOS. TRADOC has organized its schools along the lines of the ITRO 
modal for !SD, with separate- directorates or divisions for different ISD functions. Thus, 
for t ample, a task list developed in a Training Analysis Division of a Directorate of 
T/ «i;.;ng Developments would be hnnded off to a Course Development Division for 
the development of training, and the resulting training would be evaluated by the 
Directorate of Evaluation. (The exact arrangement varies from school to school.) The 
responsibility of each division, however, is primarily the generation of one or more 
products for the Enlisted Personnel Management System (EPMS) tor use outside the 
school. These include lists of training tasks for each MOS (Soldier's Manuals and 
Commander's Manuals) and evaluation instruments and training materials based on these 
lists (Skill Qualification Tests, Army Training and Evaluation Program tests, Training 
Extension Course lessons). 

Resident training, not being one of these EPMS products, is not necessarily based 
on the same task lists. It is generally developed by instructors in training departments, 
independently of the analysis and design activities of the various divisions in the school. 
The activities of these divisions, then, do not usually lead to the systematic revision 
of training, at least initial training, for an MOS. When ISD is applied to resident train- 
ing, it typically follows a decision to convert a course to self-pacing. In such cases, 
an ad hoc team is sometimes assembled to carry out the process. 



NAVY 

For initial "A" School technical training, the Navy has centralized ISD activity 
at Instructional Program Development Centers (IPDC), under the Chief of Naval 
Education and Training Support. Two centers have been established; more are 
planned. One course developed at Ik DC San Diego has already been turned .*> 
Chief of Naval Technical Training (£NTT) for implementation. 

To the extent that the IPDC approach is pursued, the responsibility for design* 
ing and developing "A" School courses will be removed from the schools, which will 
be responsible only for conducting the training. This arrangement has ostensibly been 
-chosen because training technologists needed to support ISD cannot b° made available 
everywhere training is conducted. Centralization of training development also pre- 
sumably permits more effective control of {he ISD process. Whether such control , 
is achieved remains to be seen. The IPDCs are staffed primarily by civilian education 
specialists and instructional materials developers. Subject matter expertise is provided 
by the instructors of the existing cdurses, who also, to a large extent, determine what 
tasks should be trained. 

Since the IPDC approach is only beginning to be implemented, most Navy "A" 
School courses are still developed and revised at the technical training center where 
they are taqght, by course instructors and civilian education or training specialists. A 
Curriculum and Instructional Standards Office is typicpily responsible for coordinating 
approval of the course by CNTT. More advanced training at Naval Air Maintenance 
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Training Detachment* and other trairing centers under CNTT is also locally developed 
and subject to CNTT approval. An analogous situation exists for training conducted 
by the Atlantic and Pacific Fleet Training Commands. 

Another erea of major ISD effort in the Navy is aircrew training. The Naval 
Training . Equipment Center (NTEC) has sponsored several large-scale analysis and 
development projects by civihan contractors, for new or updated aiicraft (one aircrew 
training squadron has also undertaken its own effort, patterned after the contractor- 
developed procedures). In some cases the projects have been limited to analysis and 
design of training. When instructional materials have b*en developed as well, the 
emphasis has been on self-paced audiovisual and printed media to replace much of 
the instruction that was formerly conducted by a one-to-one tutorial or in the group- 
*paced lecture/demonstration mode. NTEC has also developed specifications to guide - 
and control future ISD efforts conducted by civilian contractors. 1 

MARINE CORPS 

The Marine Corps has not ipfetituted a new form of organization to implement' 
ISD, but rather has concentred on teaching ISD procedures in its instructor training 
courses Eormal trainmg-ttcleveloped at the resident school where it is conducted, 
primarily by course instructors. (Civilian education specialists also participate in 
course design to some degree, especially in the preparation of course control docu- 
ments for approval.) The Marine Corps policy is to require all instructors to apply 
ISD whenever they are developing a new course, but not to revise existing training 
that had been developed in accordance with the Marine Corps* earlier nine-step "sys- 
tems approach" model fof training development. 

The extent of application of ISD has been quite limited. There are instances, 
however, in which training has been redesigned or developed using some of the ISD 
steps. The personnel involved ranged from one instructor or education specialist to an 
ad hoc team of several instructors, sometimes organized under the name of a Course 
Content Review BoardT^Vhen such teams are used .to design training, the actual 
authoring of instruction (generally lesson plans) is still generally accomplished by the 
instructors who will -use it. 



AIR FORCE 

Technical training by instructors in the Air Force is developed at the resident 
schools where it is conducted. Instructor training has therefore been revised to 
include ISD techniques. In addition, an ISD Specialty has been added to the Air 
Force job structure. ISD Specialists and civilian education specialists provide guidance 
to the instructors who are developing or revising their courses, and they coordinate 
approval of the course objectives (Program of Instruction) by Air Training Command 
(ATC). At the time of this report, the future of the ISD specialty is in some doubt. 

The application of ISD to Air Force technical training is greatly conditioned by 
certain components of the existing training development system. An occupational 
measurement squadron is responsible for providing job task lists to schools. For each 



' Mi'itary Specification MILT-29053, Training Requirements for Aviation Weapon Systems: 1977. 
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course, school* are required to publish a Course or Specialty Training Standard that 
lists tasks selected for training. A list of/ training objectives must be published in the 
form of a Program of Instruction. To^ large exte tt, the application of ISD in tech* 
nical training consists of satisfying tljMfe documentation requirements. , 

The application of ISD to aircrew training is more centralized. For each aircraft, 
trainir ? is developed at a single site, although it may be conducted at several. As in 
Navy flying training, nugor ISD efforts are being undertaken to develop individualized 
(usually audiovisual) instructional materials an J\o substitute simulator hours for flight 
time. The typical approach is the use of an ISu^eam, either from an ISD squadron 
or as a part of the flying training squadron or wing where the training is being devel- 
oped. Although team members may be aircrew personnel assigned to ISD duties, they 
are not the flight line instructors who will teach the course. This degree of separation 
of functions has the potential to increase the degree of objectivity in the job and train 
ing analysis, but also presents the prohlems of communication and resistance inherent 
in any effort to separate training development from implementation. 1 



SUMMATION / 

No particular method of organizing for ISD (centralization or decentralization, 
separation of functions or unitary approach) was found to promote or hinder its 
application. To a certain extent this finding reflects several difficulties encountered 
in trying to assess the relative effectiveness or efficiency of alternate approaches:. 



(1) None of the training design efforts examined followed ISt> procedures 
closely enough so that the adequacy of the resulting training could be 
considered a measure of the adequacy of ISD, much less of a particu- 
lar approach. 

(2) Data on the cost of applying ISD, on which to base comparisons of 
efficiency, were seldom available. 

(3) The ISD efforts that were. examined cover a range of types of training 
in different content areas. Measuring the effectiveness of alternate 
organizational approaches would require that the costs and effects 
associated with the way the ISD process has been structured be identi- 
fied and isolated from the costs and effects associated with particular 
training methods or jobs. 



These difficulties notwithstanding, the research team's conclusion after examining 
57 courses is that whether or pot ISD procedures are closely -followed is not a func- 
tion of the manner in which people are organized to apply it. At least within 9 the 
range of approaches represented in the study, none was found to be superior. 

It was also found that separating ISD functions does not of itself insure a high 
degree of objectivity. WhenHhe effort is made to insulate one ISD activity from the 
influence of another or from past practice, by handing off ISD products from one 
party to another, the oroducts are often modified to suit those who receive them. 
This applies to hands-offs both among instructional developers and from developers 
to instructors. 



*See Miller, Ralph M., Swink, Jay R. t and McKenzie, James F. t Jr M instructional Systems 
Development (ISD) in Air Force Flying Training, Air Force Htftnan Resources Laboratory 
TR 78-59, 1978. W 
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ISD U typically performed by a imall team. Even when tome other form of organi- 
zation has been formally established to carry out I8D, there is a tendency to revert to 
a small team. In the Marine Corps and in flying training the team approach is standard; 
in other instances teams are often assembled ad hoc. 

, ISD is generally performed by- instructors. Of the 57 training design efforts studied, 
instructors were primarily responsible for G9. Moreover, even when analysis and design 
are accomplished by others, the actual authoring of instruction is often performed by 
these who teach tne course. 
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Chapter 5 
FIELD STUDY OF ISD APPLICATIONS 

FIELD STUDY OF COURSES 

Information about how ISD is being applied was obtained' by examining the proce- 
dures used in developing or revising 57 courses at 33 organisations in the four Services, 
draining developers who had participated in course design described how training had. been 
developed and implemented in each case. The initial identification and screening of instal- 
lations to be visited and courses to be examined was made on the basis of information 
obtained & the questionnaire survey described in Chapter 3. The strategy used in identifying 
sites was to seek out courses that provided the most complete representation of the I8D 
process. This apprrisal was verified in phone conversations prior to a visit and again upon 
arrival at the site. Occasionally a course Chat, had been previously selected was deleted, 
and one that developers felt represented a better example of their best efforts to employ. 
ISD was substituted. The sample of courses that was examined represents the best txam- 
pies of ISD that were identified. Thus, the sample is exemplary rather than typical. 

In many cater several persons provided information about a given course. Each step of 
the design process that had occurred was first identified; then the procedures and products 
for each step were described and d i scussed in detail. Each interview covered only a single 
course and how it was developed. This approach resulted in a body of data consisting of a 
collection of spectffc* rather than a collection of general procedures. A structured format to 
the interview provided for cross-questioning, to check on the accuracy of the information. 1 
In almost all instances, however, interviewees proved to be quite candid in their descrip- 
tions of what they had and had not done* 

As indicated in Table 10, courses were examined in each Service. Combat/air crew, and 
technical training, both maintenance and non-maintenance, are represented. The sampling 
of courses, however, is not related to sucli factors ss combat importance or input density. 
Instead, an effort was made to include examples of as wide a range of types of training as 
practical.. Intentionally avoided were courses with a strong educational and informational 
orientation, non-job-specific courses, and othc in which it is difficult to identify specific 
behavioral objectives (e.g., Service academies, leadership, organisational effectiveness). 

The remainder of this chapter contains a step-by-step description of the way in which 
the ISD' process is being applied*. It is organized like Chapter 2, discussing each ISD step in 
turn. Following a summary description of each step 2 are a table listing prerequisites and 
procedures for carrying out the step; statistics oh the number of courses in which the pre- 
requisites were present and the procedures were carried out; and a general description of 
the way in which the step has been applied. The chapter concludes with a general discussion 
of how ISD is being used. In Appendix B, the data on the individual steps summarized for 
all Services in this chapter are broken down by the individual Services. 



1 Copies of the interview guide can be obtained from HumRRO, Western Division. 

2 Each step, previously described in Chapter 2, has been summarised so that the present chapter 
can be read independently. The reader who has read Chapter 2 may wish to skip over the introductory 
description of each step. 
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> TabitIO 
Count! txtmintd 



— aBMX 

• Short Ranga Air Dafansa Minilt Crawman 
' - Air Dalanta School, Fort Bliss, Taxas 

Infantry Off ictr Advanced 

- Infantry School, Fort Banning, Georgia 

• Field Artillary Crewman 
, - Field Artillery School, Fort Sill, Oklahoma 

• Atrial SurvaillarJ^Sansor Repairer 

- Intelligence School, Fort Huachuca, Arizona 

• Tracked Vehicle Mechanic 

- Armor School, Fort Knox, Kentucky 

Improved Hawk Fire Control Repairman 

• Nuclear Weapons Electronic Specialist 

- Missile and Munitions School, Redstone Arsenal, Alabama 

NAVY 

• Avionics Technician t 

• Basic Electricity and Electronics 

- Naval Air Technical Vaining Center, Memphis, Tertnesse^ 

• Aviation Storekeeper 

• Aviation Maintenance Administrationman 

• Personnelman 

- Naval Technical Training Center, Meridian, Mississippi 

• Propulsion Engineering - 

• Fire Control Technician 

- Service School Command, Great Lakes, Illinois 

• Radioman 

• Internal Communicotionman 

~ Instructional Program Development Center, San Diego, California 



(Continued) 



Teble 10 (Continued) 
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• F-4J Electrical Syrtemt Organizational Maintenance * 

- Naval Air Maintenance Training Detachment 1024, MCAS Beaufort, South, Carolina 

e AN/APM 226 Module Analyzer Teat Coneole 

- Naval Air Maintenance Training Detachment 1003, NAS Oceana, Virginia A 

• SH2F Airframe Maintenance 

» 

- Naval Air Maintananca Training Datachmant 1007, NAS Norfolk, Virginia 4 

• Hagan Automatic Boilar Control Conaola Oparator 

• Storekeeper Supply Afloat Indapandant Duty 

• AN/SPS 48 Haavy Radar Maintananca 

- Fleat Training Center, Norfolk, Virginia 9 ~ , 

• Nuclaar Weapons Advancad Maintananca n v 

- Nuclaar Weapons Training Group Atlantic, Norfolk, Virginia 

• Surfaca Anti-Submarine Warfara Officar . 

- Flaat Anti-Submarine Warfara Training Cantar Atlantic, Norfolk, Virginia 

• OK-252/BQR-16 Array & Cable Handling Group 

• MK24 Hydraulic/Support Ring Advanced 

- Flett Ballistic Missila Submarina Training Cantar, Charleston, South Carolina 

• P 3C/P 3B MOD Aircrew 

- Patrol Squadron 31, NAS iftoffet Field, California 

• F14 Aircrew 

- Fighter Squadron J 24, NAS Miramar, California 

• SH 3H Aircrew 

- Helicopter Anti-Submarine Squadron 10, NAS North Island, California 

* 

• Basic Underwater Demolition and SEAL 
* Naval Amphibious School, Coronado, California 



MARINE CORPS 

Instructional Management 
IBM 360 Systems 
Communications Officer 

- Education Center, Development and Education Command, Quantico, Virginia 

* * 

- — (Continued) 



X 



Table 10 (Continued) 



• Motor Transport Staff NCO 

- Motor Trantport School, Camp Lejeune, North Carolina 

• Food Sarvict Basic 

• Fiscal Accounting 

- Sarvica Support Schools, Camp Ltjauna, North Carolina 

» • Journeyman Combat Enginatr 

- Enginaar School, Camp Lajtunt, North Carolina 

■» 

. AIR FORCE 

t 

• Talavision Equipment Repairman 

• Tachnical Instructor 

• Sarvices Operations Officer 

• Instrumentation Mechanic: Sets Training . 

• Material FaciKties Specialist 1 

- towry Technical Training Center, Colorado 

• Dental Supervisor . 

• Optometry Specialist 

• Mental Heelth Ward Specialist 

- School of Health Care Sciences, Sheppard AFB, Texas 

• Aircraft Propeller Mechanic 

• Contract Construction Inspector 

• Reciprocating Aircraft Maintenance Specialist 
Outside Wire and Antenna Repairman 

• Accounting and Finance Officer 

- Shepperd Technical Training Center, Texas , 

• E-3A Avionics Instrument Specialist 

• E-3A Electrical System Specialist 

- Field Training Detachment 413, Tinker AFB, Oklahoma 

• E-3A Weapons Director 

• , E-3A Airborne Computer /Display Maintenance Technician 

- 552 AWAC Wing, linker AFB, Oklahoma 

• AQM-34M Missile Systems .Maintenance Specialist 

- Field Training Detachment 512, Davis-Monthan AFB, Arizona 

• A-10 Pilot 

. _ 4444 Operations Squadron (TAC), Davis-Monthan AFB, Arizona 

• H-53 Combat Rescue Aircrow 

- 1550 Aircrew Training and Test Wing (MAC), Kirtland AFB, New Mexico 
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STEPS OF PROCESS 

NMd/Ditertpancy Analysis 

* > 

ISD originates with the identification of a need to develop or revise instruction. 
Instruction must be developed if the human performance elements of a new job or . 
weapon system require training. Instruction must be revised if existing training is not 
sufficiently effective and economical.' Prerequisite conditions for determining need are 
the existence and maintenance of procedures for detecting indications of potential dis- 
crepancies. When a potential discrepancy is identified, additional information is gathered 
as needed for further confirmation and definition of the problem, and to suggest the 
appropriate locus for corrective action— that is, where the ISD process should begin, and 
the extent of ISD that is appropriate to the problem. 



Tsbltll 

Need/Discrepancy Analysis: 
Inctdencn of Prerequisites and Procedures in Courses Examined (N-57) 



JS* 

Prerequisite 


Present 


A bum 


Not Applicable*/ 
Not Determined 


1. Information is available about possible dis- 
crepancies between training and field . 
requirements. 


31 


2 


24 


Procedures 

1. Analyze initial and supplementary 
information. 


Pont 
24 • 


Not Dona 
22 


Not Applicable 0 / 
Not Determined 

11 ' ' 


2. Identify and specify discrepancy. 


23 


23 


11 


3. Specify ISO entry point and boundaries 
of redesign process. 


19 


27 


11 



'Not Applicable prerequisites occur in instances where their ttatut it moot <e«., prerequisite states the location 
in ISO process whereNtep is to occur, but step has not been performed). 

b Not Applicable procedures occur in ISO efforts thet wete examined jh progress and for which a given step was 
not yet appropriate. 



In about one-third of the courses studied, it was found that training development 
had been initiated in rerponse to an identified training deficiency, and that the entry 
point for ISD had been specified accordingly. In most cases, however, the deficiency 
was simply the lack of any training program for some/newly introduced equipment or - 
a new job specialty. Only five of the development efforts (9$) had been undertaken in 
response to indications of poor training or job performance, such as failed inspections 
or high attrition. In 11 courses, it could not be determined whether the identification 
of a training deficiency had led to the requirement to redesign existing training. Train- 
ing developers interviewed did not always know what circumstances preceded their 
own efforts. 
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The low incidence of cues in which specific training -defi^lenclfet were identified 
and remedied through the ISO process may be partly due to the manner of selecting 
the sample of courses for this study. Course* which had reportedly undergone many 
ISD steps were selected ove/those which had undergone only a few; and training devel 
open on site were asked to ascribe their "most complete" ISD efforts. Both of these 
factors would tend to favor the selection of training design efforts which had* begun 
with front-end analysis, rather than those for which a later ISD entry point had been 
specified. 1 00 
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Identification of Job Requirements 

Job analysis consists of three parts: compiling a provisional list of tasks believed 
to comprise the job; verifying the accuracy of the list and adjusting it as necessary; and 
gathering task priority information. Task lists are developed in a two-phase process to 
increase their reliability. The taitial list may be generated in various ways, ranging from 
the recollections of a single subject matter expert to extensive observation of job incum- 
bents. Verification may also take different forms, ranging from review by a second group 
of subject matter experts to a phone or much broader mail survey. Ideally, verification 
includes validation with a representative sample of job incumbents. 

Task priority information is data about such factors as billets where the task is per- 
formed, percent, of persons performing, and frequency of performance. To increase the 
reliability of what are inevitably subjective estimates, this information should usually be 
gathered from a fairly large sample of job incumbents. 

Tsblt 12 

Identification of Job Acquirements: 
Incidence of Prerequisites and Procedures in Courses Examined (N-B7) 



Prerequisites 

None ? 


Prasent 


Absent 


Not Applicable/ 
Not Determined 

t 


Procedure* 

1. Construct provisional task list. 

2. Verify and revise on basis of review/ 
survey of job incumbents. 

3. Collect task-priority information. 


Done 
38 

27 
24 


0 

Not Don* 
13 

20 
23 


Not Applicable 
.Not Determined 

6 . 
10 

1 



A provisional task list was constructed in 38 of the courses studied, and in 27 of 
those cases it was subjected to some kind of verification by job incumbents or other 
subject matter experts. In the Army most of the lists were constructed by personnel 
permanently assigned to task analysis duties at TRADOC schools, as in a Directorate of 
Training Developments, in the Marine Corps the lists were generally constructed by 
resident school instructors temporarily assigned to course development duties, sometimes 
as members of an ad hoc Course Content Review Board. In the Navy the list was likely 
to be an occupational survey (NOTAP) printout, or an existing list of watch-standing 
or equipment maintenance duties. When a Navy task list was constructed specifically 
for course development, this was usually done, in the case of technical framing, by 
instructors under the coordination of a Curriculum and Instructional StandardsOffice, 
or, for air crew training, by former incumbents temporarily assigned to work with an * 
ISD contractor. Most Air Force task lists were CODAP printouts. When* list was con- 
structed by Air Force course developers, which occurred only rarely in non-flying train- 
ing, it was compiled by instructors. 

The principal instances in which job taslriBts-w**e not compiled were a few field- 
conducted, equipment-specific courses in the Navy and a v riety of Air Force courses 
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for which no occupational survey report was available. Of the development efforts 
examined, those in the Air Force had the highest (50%) of cases in which task lists 
were definitely not compiled. It may be that the wide availability of CODAP printouU 
leads Air Force training designers to consider task list construction solely the respon- 
sibility of occupational measurement personnel, The procedure specified in Air Force 
Pamphlet 50-68,. however, is to use occupational survey data along with other information, 
to build a complete task list arranged by job duties. This approach was rarely taken 
in Air Force technical training. If a CODAP printout had been obtained, the develop- 
ment of a Course or Specialty Training Standard (selection of tasks for training) was 
generally reported as the next ISD step accomplished. 

If either a NOTAP or a CODAP printout was available and reported to be used, it 
was considered a verified task list for the purposes of this study. However, personnel 
interviewed often questioned the usefulness of these occupational survey reports, on 
the grounds that the task descriptions were not suitable for deriving training objectives. 
As the recommended AFP 50-58 procedure implies, perhaps such lists are not adequate 
by themselves to serve as the basis for training development. Examination of some of 
*he NOTAP/CODAP lists used to develop courses in this study revealed that some items 
listed were well-defined task descriptions, while others were actually general skills or non- 
task categories of behavior. 
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Selection of Tasks for Training 

tasks identified by job analysis are classified into those to be trained and those not 
to be trained. Tasks selected form the basis of the training program; those rejected are 
not reconsidered until the final external evaluation. Prerequisites for task selection are 
a list of tasks.which comprise the job, information for establishing their importance and 
need for training, and decision rules for determining training priorities. To promote objec- 
tivity, this information and the procedures for processing it should be developed prior to 
the selection process. After identifying which tasks are desirable to train, and which tasks ' 
are most important, requirements for training resources are considered. Final selection 
is made on the basis of training resource requirements and availability— that is, cost. A 
factor which greatly influences cost is the training setting, but to avoid bias in task selec- 
tion the setting is selected separately, insofar as possible. 

' ' • s . 

Table 13 * j ' ; 

* • " 

Selection of Tasks for Tracing: 
incidence of Prerequisites and Procedures in Courses Examined (N=57) 



Prerequisites V 


Present 


■ ■ i 

Absent 


Not Applicable/ 
Not Determined 


1 . Tasks performed in the job are listed. 


38 


13 


6 


2. Information is available for establishing 
importance of tasks and need for training. 


«« 

23 


1 

23 


f; 

11 


3. Decision rules to be applied to task 
information ace available. 


* 

7 


40 


10 


Procedures 


Done 


Not Done 


riot Applicable/ 
Not Determined 


1 Apply decision rules to information for each 
task to determine training priorities. 


5 


41 

» 


11 


2. Select tasks for training on basis of training 
priorities and resource availability. 


6- 


40 


I 

I 11 



/ -. 

Although a job task list of some kind was available in 38 of the courses studied, 
information for establishing task importance was available in only 23, and decision 
rules for selecting tasks for training were applied to this information in only fjve. More- 
over, such decision rules as were followed were rudimentary at best. In one Jase, the 4 
rule of thumb was to train any task which at least 5% of 'incumbents jjerfornfiedjin 
another it was 30%; in a third, it was tasks jud^ti critical to mission accomplishment. 
In a fourth instance, tasks were generally chosen if performed bf 60% or more' of 
incumbents, or if critical or difficult. • * V" ' 

Overall, the selection of tasks for training in the four Services was not characterized 
by deliberate determination of task selection criteria and the subsequent selection of 
tasks on the basis' of task priority information. When priority information was collected* 
it was rarely employed in any systematic fashion. When training designers rtported that* 
such information was used in selecting tasks, it was common to find that it had been 
furnished by the same subject matter. experts who were making the task selection, and 
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at the same time. Even when 'the effort was made to collect Usk^priority information 
tiom Job incumbents independently, respondents were, often asked to indicate the cri- 
ticality, difficulty, frequency, and so forth of a task and at the same time whether it 
should be trained-thus confouriding job analysis with task selection, and losing the 
advantages in objectivity of keeping the activities separa£e. Under these conditions, one 
task might be selected for its frequency, and another for its difficulty. Applying the 
same criteria to all tasks, or certain criteria to particular classes of tasks, rarely,, if ever, 
ocpurred. Tasks were seldom ranked in importance to allow the most efficient alloca- 
tion of available training resources or to aid in determining what those resources should 
be. Thus, any potential of the ISD approach for determining training resource require- 
ments acc<ffding to task importance was not generally being realized. 

Aside from the absence of decision rules, other practices tended to make task selec- 
tion less than systematic. Often, the original job task list was not maintained in readily 
usable form. Once tasks were selected, it was difficult to determine where they had 
come from. In other cases, task selection began before job analysis was complete, or was 
substituted for it altogether. .This may have been due to a natural reluctance to expend 
effort delineating tasks felt to be unimportant, or to management emphasis on records' 
of tasks selected -u training (e.g., Training Standards, Soldier's Manuals) rather than job 
task inventories. 

Records showing which job tasks had been selected and which rejected were seldom 
available, and records showing why a given task had been selected, hardly ever, tasks 
were corrfmonly combined, subdivided, or otherwise altered in the process of going from 
job requirements to training requirements, especially -when the task list was an occupa- 
tional survey report. Often the only evidence of which tasks* had been selected was a set 
of training objectives, each of which might cover several tasks, or only part of one. fnv 
the absence of records to the contrary, it is possible that the tasks "selected" for training 
in a given course were the only tasks* cbnsidered-that is, selection woufd not have 
occurred. 

In the Army, tasks were selected by personnel assigned to job analysis duties in the 
Directorate of Training Developments at a, TRADOC school, who were responsible /or 
developing Soldier's Manuals. After tasks had been selected for this purpose, the same 
task lists were forwarded to other directorates or divisions within the school to serve as 
the basis for TEC lessons, Skill Qualification Tests, resideht training, and other training 
elements. It was found that the selection of iasks made by those responsible for job 
analysis was not always accepted and used by those to whom it was forwarded. This 
was especially likely to happen , in the development of resident courses, where in some 
instances the activities conducted at the "front end." of the ISD process had little or no 
^connection with what was taught in the classroom. This situation was not universal, 
(across the six Army schools visited, but it was typical. 

It should be noted that the Army is currently emphasizing training and evaluation 
products for use in operational units, rather than in schools (Soldier's Manuals, TEC 
lessons, SQTs, etc.). With respect to resident training, more emphasis is being given to 
changing training methods than to determining content. ISD activities were being applied 
to resident training primarily when courses were being converted to self-pacing. 

Since task selection in the Army is performed at the school which is proponent for 
doctrine in a particular MOS, command. approval is inherent in the selection process. In 
the Navy, Marine Corps, and Air Force, tasks selected for training (usually in the form 
pf training objectives) are typically sent to a higher command for approval. In the Navy 
and Air Force, where training developed at one site may be conducted at others as well, 
and graduates of a course may be assigned to different commands, it is common to 



seek the concun^ce of the other units or commands invoked. In the Air Force, a 
training review conference ("scrubdown") is sometimes convened, bringing representatives 
of receiving commands together with representatives, of ti\e school where initial training 
is developed and implemented. 

Efforts to obtain agreement on training content represent an apparent departure 
from ISD methodology, which suggests the use of rigorous decision' rules, rather than 
the achievement of consensus. Personal judgment may, however, be the only means for 
task selection now available. As noted in Chapter 2, there are currently Vio known cri • 
teria of to'tal system effectiveness by which to determine task selection criteria. Perhaps 
the almost universal failure of training designers to select tasks systematically according 
to predetermined criteria i» in recognition of how arbitrary any set of task selection rules 
niust be. Even when given a mathematical formula to apply to available information, as in 
Air Force Pamphlet 50-58, no one reported using it. * „ 

In the absence of decision rules for task selection, there may nevertheless be some 
•value m requiring task priority information to be collected by survey, or even "developed" 
when tasks are selected. The mere availability of such information may make the per- 
sonal judgment of training designers' more informed, and the requirement to justify task 
selection on some grounds may make the selection more thoughtful. 
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Analysis of Tasks 



Taslj analysis is a description of when and how, within the job environment, perform- 
ance of a task is required. It consists of a specification of conditions of performance and 
initiating cues, behavioral elements, and standards of performance. Information generated 
dunng task analysis is used later in constructing Job Performance Measures and developing 
objectives. The prerequisite for task analysis is the identification of tasks to be analyzed. 



Table 14 



. Analysis of Tasks: 
Incidence of Prerequisites and Procedures in Course Ekamined (N«57) 



Prerequisites * 

1 . Tasks selected for training are listed. 



Procedures 



1. For. each task specify for the job 
environment: 

. - Conditions of performance. 

- Behavioral elements.' 

- Standards of performance. 



Present, 
41 



Done 



16 
31 
19 



Absent 
10 

Not Done 



29 
13 
28 



Not Applicable/ 
Not Determined 

6 



Not Applicable/ 
Not Determined 



12 
• 13 
10 



In the 41 training design efforts in which t As were identified for training, bfehavioral 
elements were identified in about three-fourths (31 courses), while task conditions and 
standards were identified in less then half. The greater incidence of identification of 
behavioral elements was partly due to their being listed in training objectives, or instruc- 
tional materials. The Army and Marines account, proportionally, for nearly all explicit 
task analysis. In the Army, task conditions, elements, and standards are listed in the 
Soldier's Manual; in the Marine Corps they were found in task analysis worksheets. In 
the Navy and Air Force, the identification of task conditions and standards as distinct 
from training conditions and standards was infrequent. >The approach generally followed 
was to specify conditions only as required for course 'control documents, with<Jht first 
having determined the standards and conditions of job performance. Two of the four 
instances of explicit task analysis found in the Navy were large-scale ISD .efforts at, the 
Instructional program Development Center, San Diego. 



Construction of Job Performance Measurtt ' 

Job Performance Measures may be constructed for each tank that has been selected 
for training to serve as a means both of keeping training faithful to job requirements and 
of evaluating training and, the design process, to maintain fidelity to job requirements, 
training objectives can be derived directly from Job Performance Measures. To evaluate 
training design, Job Performance Measures are administered to persons in training, :>r after 
th^y have reached the job v To serve their purposes, Job Performance Measures mu.t be 
constructed before objectives or achievement tests are developed. Not all models of ISD 
include the'construction of Job Performance. Measures. 

* i 

f Table 15 

Construction of Job Performance Measures:. 
Incidence of Prerequisites and Procedures in Courses Examined (N-67) 

* — . h. , : ^ \ 



Pr»renni«itiK ' Not AppliHWe/ 

Prerequisites Prwent .. Abient t Not Determine*! 



43 



1. Tasks selected for training are listed? 12 2 

2. Training requirements for these tasks 

have not been identified. 8 6 43 



^Procedure* ' ' ~ Not Applicable/ 

frroceoures i Done Not Done Not Determined 



1 . Construct a test for measuring the per- 4 , * 
formance of each task selected for training. 11 36 10 

2. Validate each Job Performance Measure to ' n 

* insure that it predicts task performaoce. 3 38 *16 



• < 



Job Performance Measures were developed iri 11 courses, and validated, or verified 
in three. In all cases but one, however, these measures were simply performance-oriented 
achievement tests constructed for within-course use. In only one instance, IPDC San 
Diego, were JPMs .constructed and at least partially validated in the field, in accordance 
with the ITRO model procedures. No other case was^ found in which training designers 
constructed task measures early in the training design process to serve as a means of 
evaluating subsequent course design and implementation. Training designers reported in 
some cases that they did not* have sufficient resources to attempt to constructor validate 
JPMs; the IPDC itself discontinued the attempt after its first course development effort, 
citing the considerable time and expense required. The remainder of the courses that 
were classified in this study as using JPMs generally incorporated exercises performed on 
actual equipment or full-scale training devices- maintenance simulators, mock warehouse, 
aircraft, and so forth. ► , . . 

c 

/ 
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Selection of Setting 



Every task that has been selected for training must be assigned to a .particular location 
and situation in which training will occur. A prerequisite to carrying out this step is the 
freedom to assign tasks to any of several settings. Andther is the availability of information 
about the costs of training in the alternative settings. This step involves an interface with 
other components of the operatipnal system, beyond the training subsystem. To assign 
tasks to different settings in such a way as to maximize total system effectiveness requires 
knowledge of how alternative assignment patterns woukj, affect the total system. 

Table 16. 

Selection ,of Setting: 
Incidence of Prerequisites and Procedures in Courses Examined (N-57) 




Prerequisites 

. 1 . Tasks can be assigned to any df 
several settings. 

a 2. Information is available on cost&of training 
in different settings. 

3. Information is available on effects of 
training in different settings on total sys- 
tem effectiveness. 



Present 

18 
0 



Absent 
23 
0 



Not Applicable/ 
Not Determined 



16 
57 



57 



Procedures 

1 . Assign /»ch task or group of tasks 
to its appropriate setting. 



Done 



16 



Not Done 



37 



Not Applicable/ . 

Not Determined 
» — 



4. 



In only 18 courses were training design«8| free to assign tasks to any of several settings. 
Usually they were concerned 1 only with the setting in which they themselves were located 
and for which they were developing training. In the Marine Corps no instances were found 
in which training designers assigned tasks to other than resident schools] and in both the 
Navy and the Air Force this occurred in only about one-third of the applicable courses. 
Although training developers in the Army were free to assign tasks to different settings in 
three out of 'four applicable cases, in one of these the procedure was simply to assign every 
task to, a* resident school unless it was impractical to teach it there. 

In almost all cases, setting selection was actually synonymous with task selection; 
personnel selecting tasks for trainmg were almost always selecting them for one particular 
setting, fhe statement in the AFP 50-58 model that course developers would seldom be 
in a position to designate training settings (see p. 32) was fqunckjto be closer to actual 
practice than the situation depicted by the ITRO model, in 'which designers pre free to 
develop job performance aids or on-the-job training as alternatives to school training. An 
exception to the general case was- the practice in some Navy .training design efforts of desig- 
nating "A" School, "C" School, formal on-the-job training, or other setting for each job task. 
Since personnel interviewed were cognizant only of how .training had been developed for 
their own setting} it was not determined whether training was, actually developed for these 
other settings as well. ' ^ 
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Development of Training Objectives and 
Objectives Hierarchies 



Training objectives are descriptions of what a trainee is expected to be able to do 
following instruction. Their development represents a shift in the ISD propess from Che 
job to training. Developing objectives involves serveral steps: 

• Deciding how closely capabilities for task performance at ^he conclusion 
of'training shall match the requirements of the job, , 

• Specifying dependent and coordinate relationships within tasks (terminal -• 
objectives), among subtasks (intermediate objectives), and mediating skills 
and knowledge ("intermediate objectives}. 

• Deciding, based on estimates of the abilities of entering trainees^ what ter- 
minal objectives must actually be developed during training, and what 
knowledge and basic skills must be provided to mediate learning and . 
performance. 
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Table 17 

Development of Training Objectives and Objectives Hierarchies: 
Incidence of Prerequisites and Procedures in Courses Examined (N«67) 



4 



Prerequisiti 

1 . Tasks selected for training are listed. 

2. Information is available about training 
constraints that make it necessary to 
modify task requirements. % 

3. Information is available about how modi- 
fication of task requirements will affect 
training efficiency. 

4. Estimates of capabilities of entering 
trainees for learning and performing each 
task are availably. 

Procedures 

1 . Specify task requirements (behaviors, 
conditions, standards) for training. 

2. Perform hierarchical analysis of tasks to 
identify intermediate training objectives. 



Preient 
44 



Done 



51 



Absent 



14 



Not Done 
1 . 
39 



Not Applicable/ 
Not Determined , 

6 



57 



57 



50 



Not Applicable/ 
Not Determined 



Training objectives were developed in nearly all cases (SI courses). The use of * 
behavioral abject! ves to describe training content is thus the single most unifying charac- 
teristic bf the courses studied. In some eases, the very purpose of course revision was to* 
add property worded objectives (e.g., observable, measurable) to course control documents. 

The relationship of objectives to job tasks, however, varied considerably. In several * 
cases object Ives were develpped in the absence of any specification of a set of tasks 

n ■ 



selected for training. In the remaining cases it was common for one objective to repre- 
sent several tasks, or vice versa, or both. In Navy and Air Force technical training this 
xps -standard practice, with a 'single* objective sometimes being listed as pertaining t6 a 
dozen or more different tasks. It was often unclear what rules; if any, hadF governed the 
• derivation of objectives from a single task, or the representation of so many tasks by 
One or several objectives. In many cases personnel interviewed did not report the use 
of any systematic procedure. Thus, although there were often records showing which' 
objectives went with which tasks, there is reason to question whether (he objectives, had 
actually been derived from the tasks. * 

In only one-fourth of the courses studied was there an effort to develop a hierarchy ' 
to derive objectives. In the, great majority of cases, no systematic attempt was made to r 
analyze tasks or other objectives to determine what must be learned in order to master 
them. Moreover^ in the cases in which some hierarchical analysis was judged to have 
occurred, an actual outline or diagram of the relationship of objectives to tasks of other 
objectives was usually absent. Rarely had trainers deliberately estimated the capabilities ' 
of entering trainees in order to determine the limits of hierarchical analysis. . • * 

^ The ISD process is. partly a means of insuring that training will not exist for its own *' 
sake, but that mastery of 'training requirements will lead to successful job performance. 
The ISD procedure of deriving terminal training objectives directly from tasks, and inter- 
mediate or enabling objectives in turn, is intended to preserve this close relationship. In 
practice, this one-to-one* correspondence between job tasks and terminal objectives was 
the* exception rather than the rule. Although there is nothing inherently unsystematic 
about deriving several objectives from the same task, or combining similar tasks, the 
typical lack of any corresponding methodology for doing, so suggest* that objectives were 
often matched to tasks after training content was already known, rather than as the .means 
of determining that content.. 

It appears .that in many cases— perhaps the* typical case-objectives had been written 
to "cover" subject matter currently taught or being added to a course. Under this con- 
dition, training objectives would have^een from training content, rather than content 
from objectives.. Further support for this conclusion, in addition to the fact that per- 
sonnel interviewed often could not specify the derivation process, is the fact that objec- 
tives often were written simultaneously with instruction and by the same persons. In 
at leasj one instance, the objectives said to be based on job analysis were incorporated 
without any changes in what was taught. 

Job conditions were specified in only 16 of the courses studied, and job standards 
in 19, compared to the 51 courses for which objectives were written. Training conditions 
and standards, then,. were usually specified in the absence of explicit job conditions and 
standards. As mentioned, this practice was found more frequently in the Navy and Air 
Force. Its consequences may be trivial,' if training designers are aware of job standards 
and conditions and r develop objectives accordingly. Just as easily, however, it could lead 
to establishing training standards that are unnecessarily high, or so low that training is 
inadequate—the very conditions tfiat the systems approach is designed to avoid. 

To the extent that objectives are developed along wjth instruction, as they often 
were, the strong possibility arises that training standards will sim t ty be written to reflect 
training content. As -the data -show, it was. not determined how much training designers 
knew about the 4 extent ,of changes in job standards or conditions, or how such changes 
might affect training effectiveness. In the absence of specified job standards or conditions, 
of course, the questions are moot, tyhen task analysis had been accomplished, some 
personnel interviewed were aware that their 'training standards differed from job standards, 
but no one reported any knowledge of the possible effects of the change. 
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Development of Achievement TMtt 



V 



Objective-referenced achievement tests are used to determine the capabilities of the 
entry population, to determine the effectiveness of training, and to diagnose student per- 
formance during training. A particular requirement of ISD is thai tests be developed 
directly from objectives, rather than from the content of lessons, to insure that they 
measure the mastery of job objectives, not merely-fining content. 



Table 18 



Development of "Achievement Tests': 
Incidence of Prerequisite and Procedures iq Courses Examined (N*57) 



Prerequisites 



1 . Training objectives have been specified. 

2. Instructional materials have- not 
been developed. 

Procedu res 

" & 

1 . Determine appropriate types of tests 
based on characteristics of objectives. 

2. Construct tests to assess attainment 
of objectives. 



Pretent 


Abttnt 


50 


0 


16 


< 

31 


Done % 


Not Done 


0 


6 


27 


18 



L 



Net Applicable^ 
Not Determined 



. 10- 

Not Applicable/ 
Not Detained 

i 51 

12 



istritc 



In 27 cases, tests were constructed to measure every terminal objective, and in the 
-other cases testing was generally extensive. A widespread practice was the use of criterion 
referenced testing in place of norm-referenced testing, a conversion that was often man- 
dated by Service- directive. ^Criterion-referenced" generally , meant that scoring methods 
such as the following ^rere employed: setting a relatively high percentage of correct 
responses for passing a knowledge test;' setting a low limit of permissible errors (often 
zer6) on performance tests; scoring each unit of a test on an "all or none" basis; and - 
.requiring trainees to pass alt tests, retaking them as required. 

Since* Service policies discourage high failure ratej from schools, the selection of 
criteria obviously involves considering the norm-what the trainees can be expected to 
learn in a reasonable time, for tests to be strictly criterion-referenced, the criteria for 
satisfactory performance would be whplly determined by job requirements, and training 
would be increased as necessary to meet the criteria. In the courses studied, these con- 
ditions did not prevail. 

The most significant finding with regard to ISD application is that tests were con- 
structed before instructional materials in only one-third of the applicable cases. Thus 
there was usually no independent criterion by which to determine whether mastering ' 
the course subject matter was related to mastering the course objectives. This applied 
especially to knowledge tests, which were hardly ever reported as having been based 
directly on training objectives (the objectives themselves often simply stated the 
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requirement to pass si written test), thus leaving to those who developed lesson plans or 
other instruction the actual determination of what the trainee must learn. •. 

It was not determined in each-case whether test developers had considered what 
types of tests or test itenwwould be appropriate to measure the objectives. Many triiners 
reported that objeetiveswere classified as either performance of knowledge; and that tests- 
were developed accordingly. Only in rare cases, however, did personnel report they 
used a finer classification of objectives in developing test items (e.g., whether "behaviors" * 
4tfe skilled or unskilled, whether "knowledge" requires the application of concepts or rules 
, or the recall of facts). , * 



\ 
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' .> Identification of Cntry Behavior 

When tests have been developed, earlier estimates pf trainee entry behavior can be 
verified. The derivation of objectives Wd beenr based on these estimates'. Since trainees 
. are not generally" available at this time, a determination of entry behavior is made by 
measuring performance of a group that is representative of the entry population in apti- 
tude, experience, previous graining, and other relevant attributes. 

Table, f9 ^ 



Identification of Entry Behavior: 
Incidence of Prerequisites and Procedures in Courses Examined (N-67) 



• 

Prerequisites 4 ^ 


Present 


Absent 


Not Applicable/ 
Not Determined 


^ 1 , Objectives have been derived through ■ 
hierarchical analysis of tasks. 


11 


38 » 


8 


2. Tests are available to measure gbjectiyes. 


45 . 


0 


12 


Procedures ^ i 


Done * 


t 

Not Done 


Not Applicable/ 
Not Determined 


*1 . Identify sample that is representative 
of tramees^ 


• 

r 

» 0 


49 


8 


2. Administer testl to sample, and 
- determine accuracy of earlier Estimate 
of entry behavior. 


< 

0 


• 49 


• « 

•8 


3, Add or delete objective as indicated by * 
test results and repeat cycle. 


0 


■ » 

49 


8 



Although objective-referenced achievement testswere available in many cases, and 
hierarchical analysis had*been used to some extent iiTdevelbping objectives, no one 
reported using tests to verify the estimates or assumptions abouj trainee entry behavior 
underlying the objectives. Unless estimates were always accurate, training requirements 
were either overestimated or underestimated. If trainee capabilities were overestimated, 
the discrepancy should appear during validation of instruction. If trainee entry behavior 
had been underestimated, however, the resultant "overtraining" would not necessarily 
be revealed by any later ISD step. t 



> Classification of Objectives and 

Selection of Instructional Activities 

Objectives are classified according to the type of capability they represent, and. * 
specific activities necesarrv to provide for learning are identified. In later steps, media 
will be chosen and instructional materials developed to support these activities. » 

Table 20 

Classification of Objectives and Selection of Instructional Activities: 
Incidence of Prerequisites and Procedures in Courses Examined (N-B7) 4 



Prerequisites 

1. Information is avaltible about types 
of instructional activities appropriate 
to acquiring different types of capabilities. 


Present 
• 

" \ 

» 0 


Absent 
0 


* r 

Not Applicable/ 
Not Determined 

57 


Procedures 

1. Classify each objective or group of 
objectives according to type of 'capability. 

2. Specify instructional activities for each 
objective according to type of capability. 


Done 
8 

1 * 


Not Done 

* 

40 
48 


Not Applicable/ 
Not Determined 

9 
8 


- — - — % 


t 




• 



In only one case werejnfcructional activities reported to be specified in accordance 
with a distinct classjfic^tiorrof behaviors required in objectjjres (e.g., recall information, 
perform gross motor movement, apply a rule* display an attitude). In some other cases, 
objectives were classified in broad categories such as motor skill or knowledge. In none 
of these cases, however, were instructional activities specified in accordance with the 
classification. 

In a few instances, general instructional activities' were suggested to lesson plan 
writers, such as the use of a visual aid or the opportunity for trainees to practice. Such 
suggestions, however, were not reported being based on an analysis or classification 
of the characteristics of the objectives. 

As stated in Chapter 2, the current state-of-the-art of instructional technology does 
not provide a basis for a full procedure for specifying instructional activities, or even for 
classifying objectives. Also, the AFP 50-68 guidance on this subject appears to be 
addressed only to persons who will author instruction, not those writing lesson specifica- 
tions. If so, it would partly explain the absence of such specifications in Air Force 
ISO* efforts. . 

This does nof explain the almost universal failure of training designers to carry out 
this step in the dther Services, whose ISD models clearly recommend that instructional 
activities be specified. No general explanation emerged from interviews with the training 
designers; the step appeared rather to have been ignored. Possible explanations include the 
novelty of the step; the* lack of a requirement to produce documents showing that it was 
carried out; mcoimplete understanding of what was meant by an instructional activity; and ' 
reluctance to assume a function that has formerly been the prerogative of instructors who 
develop lessons. 
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Selection of Instructional Methods 



After instructional activities have been specified, instructional methods are selected. 
These are the ways in which trainees will be brought into contact with the instruction. 
Methods are selected to maximize the effectiveness and efficiency of instruction, given 
the particular instructional activities, the nature of the' trainee population, the setting, and 
administrative requirements and constraints. 

Table 21 



Selection of Instructional Methods: 
Incidence of Prerequisites and Procedures in Courses Examined (N»57) 



Prerequisites 


Preterit 


Absent 


' Not Applicable/ 
Not Determined 


f. Setting has been specified. 


52 


.0 


5 


2. Trainee characteristics have been identified. 


0 


1 , 


56 


3. Instructional activities have been specified. 


1 


49 


7 * * 


4. Information is available on itew the costs 
and effectiveness of alternate methods vary 
for specified settings, trainee characteristics, 
and instructional activities. 


0 


3 


4 

\ 

54 


Procedures ^ 


Done 


Not Done 


Not Applicable/ 
Not Determined 


1 Specify the methcflds of instruction to , 
be employed for each objective or group 
of objectives. 


♦ • 

51 


0 


- > 

, 6 



The determination of typ> oTpacing, group size, control of instruction, and other 
instructional choices i^iriherent in implementing any training program. Thus Table 21 
shows that method*-wejp specified in all applicable. courses (52). There was little evidence, 
bowever>that-the choice was made on the basis of the characteristics of the trainees or the* 
instructional activities, or with the knowledge of what effects alternate methods would 
have on training effectiveness or cost. In the typical, case, it was reported that no syste- 
matic procedure was used to select methods. Either changing existing methods had. not 
been considered or the option to do so did not rest with the^ personnel interviewed. 

In a significant number of training design efforts, howeier ,*a change in methods had 
been directed, and was the stated object of the course revisvSn. The most common change 
was the conversion of courses to self-pacing and individualization of instruction, notably 
in the Army, and in Navy and Air Force flying trainings Conversion to computer manage-- 
ment of instruction is another example. In both types of cases,' the decision was apparently 
made by higher command that a particular method offered advantages that were general 
enough to apply to all training covered by the directive. The basis on which the decision 
was made was not determined. 
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. Sttetion of Media* 

. ***** instructional activities and training methods have been specified, and before 
instructional materials are developed, media are, selected that will provide an effective and 
efficient means tot presenting the subject matter to the trainee. Media which satisfy the 
requirements of thp learning activities and methods are first identified, arid then costs 
determine the finaJ| selection. 

- "*\ Table 22 ' ■ 

Selection of Madia: 
Incidence of Prerequisites and Procedures in Courses Examined (N-67) • 



. Prerequisites • > j 

4 1 . ^Training designers are free to select ' 
from a range of media. > 

2. Instructional activities have been specif ied. 

3. Methods of training have been specified. 

4. Information ^available concerning the 
appropriateness of different media to imple- 
ment different activities and to be used in 
conjunction with different methods. 

5. Information is available concerning tbe 
costs of different media. ^ 



Present 




Not Applicable/ 


Absent 


Not Determined 








6 


9 




4 


45 


8 . 


39 


- * 1 


17 



57 



57 



Procedures 

1 . Determine which media will be suitable 
to implement the instructional activities 
and methods. ' , , 

2. Consider relative costs of media deter- 
mined above and select most economical 
set of media. 



Done 



"^NSrOone 



47 



48 



Not Applicable/ 
Not Determined 



8 



. T* 1 * ISD procedure of identifying media appropriate to the instructional methods and 
the stimuli inherent in the instructional activities was hardly ever followed. Instructional 
activities had been identified in only f&ur' courses. A systematic attempt to identify media 
appropriate to particular objectives, was made in only three cases, and the attempt to con- 
sider costs and select the most efficient set of media occurred in only one instance. 

Training developers indicated that in general this step, lik e the selection of instruc- 
tional methods, had simply not been attempted. AgainftitheVho cons^erftimr^tas given 
to changing existing media:, or training designers were not at liberty to do fco. Where"^, 
personnel interviewed were aware that this was an ISD step they had not carried out, the 
cost of converting to a different medium was typically cited as a limitation on their 
design options. » * 
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Analogous to the selection of method*, converting to a new medium was sometimes 
tUe primary reason' for revising a course. In such a case madia selection occurred before 
any of the* other 1SD step*. Such conversions were typically undertaken to facilitate a 
change to individualized instruction.^ . ' 

As noted in Chapte> 2, the current state of instructional technology does npt permit 
the prediction of which medium w>H be most effective in training for a given objective. 
The algorithms provided in the various ISu manuals are not sufficient by themselves to 
allow a trairhng designer to identify the most efficient medium, even if cost information 
were available and the* stimulus requirements of instructional activities tould be identified 
in every instance. t Whether training designers perceived the inadequacy of the procedural 
aids is not clear, fn any eveht, no attempt tonise them was reported. 



84 



■ ' ; ... / 

Grouping and Sequencing of Inttru 

Objectives are grouped and sequenced according to dependency and commonality 
of subject matter, and transfer and efficiency of learning. Decisions are also subject 
to practical and administrative considerations. A principal characteristic of ISD is that 
decisions about sequencing and grouping be made on the basis of learning factors as much 
as possible, rather than solely on the basis of administrative factors or past practice. 

Table 23 



Grouping and Sequencing of Instruction: 
Incidence of Prerequisites end Proceduees in Courses Examined (N=57) 



Prerequisites < 

1 Knowledge of tha effects on learning 
of different sequencing plans is available. 


Present 

1 • 


Absent ' 
5? 


Not Applicable/ 
Not Determined 

4 


Procedures 

* - . 

1 Identify corrtmonality of subject matter 
^ and anticipated transfer of learning 
between objectives. 

2. Identify degree of dependfhcy between 
objectives. , 

,3, Select overall sequencing principle(s). 

4. Group and sequence objectives. 


'Done 

A. 

. 7 

♦ 

13 
13 
51 


Not Done 

39 
29 

' £ 32 

o 


Not Applicable/ 
Not Determined 

11 

15 

'12 
6 



As with selection of methods, a determination of sequence is inherent in any training 
program, and Table 23 shows accordingly that this was accomplished in all applicable 
cases. However, a systematic attempt to base the sequence of, instruction on the degree 
of commonality, dependency, or learning transfer among objectives was reported in only , 
six cases. It may be significant that two of these six cases were Navy flying training courses 
developed by civilian contractor*. Although commonality or dependency was identified 
in some other cases as well, there waano evidence that these relationships among objec- 
tives had been used to determine the riS&st effective instructional sequence. 

In the typical case, no consideration was given to how alternate sequencing strategies 
would affect learning. In many cases the schedule of equipment and facilities availability 
was cited as the controlling factor in sequencing. In other cases, it was reported that 
the proper sequence was obvious or had followed "logical" order, which generally meant 
providing instruction on one system or piece of equipment at a time, beginning with theory 
of operation or other general information and proceeding to specific information or practice. 
If an existing course was revised, changing the existing sequence was usually not considered. 

Next to equipment and facilities availability, dependency is perhaps the most .readily 
apparent sequencing consideration. Factors such as transfer of learning are harder to esti- 
mate, and, as noted in Chapter 2 r principles of sequencing are ambiguous. It may be that 
training developers did not consttiir that a systematic procedure was necessary to discover 
what was dependent, and that any attempt to consider other factors offered little* promise. 
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dtvttopmtnt of Plan for 
Authoring ajid Managing Instruction 

A plan is prepared to guide those who will author, conduct, and manage the instruc- 
tional program. Whether in a single master plan or in separate documents intended for 
the different personnel concerned, the decisions and specifications made in previous ISD 
steps are transmitted to those who will carry them out. Decisions about administrative 
matters such as class schedules and equipment use are included. 

.*' Table 24 * 

Development of Plan for Authoring pnd Managing Instruction: 
Incidence of Prerequisites and Procedures in Courses Examined (N-57) 



Prerequisites 

1. Objectives, instructional activities, 
methods, and media have all been 
specified. 



Present 



Absent 



44 



Not Applicable/ 
Not Determined 



8 : 



Procedures 



1 . Specify the content and design of 
each lesson. 

m 2. Specify how the instruction will be 
conducted and managed. 



Done 



31 



Not Done 



15 



Not Applicable/ 
Not Determined 



11 



6 



In 31 courses, the design and content of each lesson were specified in a plan of some 
kind. The extent to which such plans had been developed prior to authoring instruction 
and had guided lesson authors could not be determined. Generally, there was little evidence 
that they had. A typical plan was. a Program of Instruction or Curriculum Outline listing 
lessons, objectives; number of hours for training, equipment required, etc. 

While objectives, instructional methods, and media were available in nearly all cases 
in Which plans Jiad been prepared, instructional activities had been identified in less than 
10% of all courses examined. Thus, the extent to which specific activities were included 
in authoring plans was necessarily quite limited. The plans available in the courses studied 
appeared to be intended more to record the content and control the conduct of trainihg 
than to provide design specifications for lesson authors. In virtually all cases, plans 
included information about how instruction would be conducted and managed (e.g., group 
or self-paced, instructor or computer managed). 



'86 



Review and Selection of Existing Materials . 

After instructional activities and media have been specified, any available existing ,' 
instructional materials are examined tq see if they match these specifications in whole or 
in part. Materials that match can be used, with resultant savings in the authoring of 
instruction. In addition to satisfying established specifications fojr instructional activities, 
methods, and media, materials adopted must also be appropriate to trainee characteristics 
and the instructional management plan of the proposed "course. 



Table 25 



Review end Selection of Existing Materials: 1 
Incidence of Prerequisites and Procedures in Courses Examined (N-67) 



Prerequisites 

1. Instructional activities have been specified. 

2. Methods of training have been specified-. 

3. Media have been specified. 



Present 
5 
49 

37 



Not Applicable/ 
Abtent - Not .Determined 

43 g 

0 8 

7 . 13 



Procedures 

1. Examine existing instructional materials 
to determine whether any- meet the 
specifications for instructional activities, 
methods, and madia. 

2. Select materials or parts of materials that 
meet the specif icafaons, or which'could be 
efficiently revised.' • * 



Done 



10 



11 



Not Done 



36 



35 



Not Applicable/ 
Not Determined 



11 



11 



Existing materials were reviewed and selected in 10 courses. In most cases^the 
materials under review were simply those of the course being revised. Instructional 
materials from, other Services or the civilian sector were reviewed in only a few instances. 

Characteristics of the planned instruction, such as methods, media, and instructional 
activities, rarely provided the basis for, the review and selection process. (Methods and 
media specifications were available in most of the applicable cases, but instructional 
activities in only fi¥<) Rather, the typical method was to inspect materials on the basis 
of general suitability to the new or revised course. 
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Authoring of Instruction 

The content of. instruction is determined by earlier decisions with respect to grouping 
and sequencing of objectives, instructional activities, methods, media, and the specification 
of consent for : dividual units/lessons in the authoring plan. The actual authoring process 
consists of preparing scripts, texts, tape/slide programs, etc. As first drafts of instructional 
materials are produced, small portions are tried out on individuals who are representative 
of the entry population. Materials are then revised to correct weaknesses, Emissions, and 
ambiguities. A distinguishing characteristic of ISD is initially to include only the bare 
minimum of instruction and. subsequently to augment instruction as needed. 

T«bl«26 



r Authoring of Instruction: 

Incidence" of Prerequisites and Procedures in Courses Examined (N-67) 



Prerequisites 


^ Present 


Absent 


Not Appficabls/ 
Not Determined 


1 . Objectives have been grouped and t 
sequenced. 


49 


0 


8 


2, Instructional activities have been 
specified. 


5 


42 


10 


3. Methods of training have been specified. 


48 


0 




4. Media have been specified, ' * 


37 


8 




5.r Lesson structure And content haVe t 
been planned. 


30 

• 


14 


13 

• 


• 

Procedures 


Done 


Not Don* 


Not Applicable/ 
Not Determined 


(. Develop lean instruction. 


11 ♦ 


26 


' 18 


2. Try out instruction on small number of 
persons representative of students. 


10 


31 


16 


3. Revise and augment instruction as 
necessary. 


s 

'5 ' 


30 


• 

22 



As indicated in Table 26 s , the sequence and methods of instruction were nearly always 
determined, prior to authoring materials, while the choice of media was occasionally left 
to lesson authors. By contrast, instructional activities were specified only about 10% of 
the time. A plan for lesson authors was available in about two-thirds of the cases. 

In only 11 courses was there an attempt to restrict the content of instruction to the 
minimum necessary. Most interviewees were not familiar with this concept. Although 
other tSD steps were often reported to have been omitted due to constraints of time or 
money, lack of awareness appears to be the important factor with respect to' developing § 
lean instruction. Anyone who reported an awareness of the advantages of keeping instruc- 
tion lean also reported trying to do so. By contrast, some interviewees reported that they 
had done just the opposite— with no apparent awareness that their approach was contrary 
to the general intent of ISD. ' • 



N,ine of the 11 cues of lean instruction were found in the Navy. These include 
both courses developed by ISD contractors fo| the Naval Training Equipment Center, 
and both courses developed by the Instructional Program Development Center, San • 
Diego— presumably reflecting a greater awareness of the requirements and purpose of ISD 
among those permanently engaged in course jdesign. Except in these cases, the virtual 
restriction to. the Navy of efforts to keep instruction lean is unexplained. ' 

The development of lean instruction was found, not surprisingly, to be associated , 
with efforts to try out and revise instruction, and with the subsequent ISD step of 
attempting to validate it. Although instructional materials were tried out and revised in • 
only five cases, three of -the five represent instances where therewas an attempt to provide 
lean instruction. Instruction was validated in less than one-fourth of the total applicable 
courses, but was valided in two-thirds pf the instances of lean instruction. The association 
of tryout, revision, and validation with lean instructional development. is quite reasonable , 
since the, practice of restricting instructional content to a "minimum requires the use of 
some parallel procedure to measure instructional adequacy . 
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Validation of Instruction. 



Instructidh is tried out under conditions that closely approximate its intended use, 
with groups of trainees representative of the entry population. Objective-referenced 
achievement tests are the primary validation, criterion.. V 



if 



TsW*27 
Validation of Instruction ; 



Incidence pf Prerequisites and Procedures in Courses Ex^inad-(N«67) 



^Prerequisites 



t. Object iv*referenced achievement tests 
are available. 



Procedures 



1 . Specify achievement test validation 
criteria (number and percent of per* 
sons \h validation sample required to 
pass tests). * 

2. Specify additional validation criteria: 

3. Present instruction, administer achieve* 
mem tests, analyze results, revise instruc- 
tion, and repeat cycle until validation 
.criteria are met. 



' Present 1 



47 



Absent 



Done 



7 
4 



30 
32 



9 



31 



Not Applicable/ 
Not Determined. 



10 



Not Applicable/ 
Not Done Not Determined 



20 
21 



17 



Although objective-referenced achievement tests were available in every case, instruc- 
tional materials were subjected to a validation .process in only nine courses. Even in two. 
of these nine cases, no criteria of adequacy were established for the validation trials. Seven 
of the nine validation efforts occurred in the Navy, including— as in the -case of developing 
lean instruction— both courses at IPDC San Diego and a contractor-developed course for 
NTEC (the other NTEC-sponsored' contractor effort had not reached the point of valida- 
tion at the time of the interview): • / 

For many courses, no self -teaching materials (tape/slide, text, etc.) were being 
produced-only learon plans to guide instructors' presentations. In such cases, the vali- - 
dation process would require repeated presentation of the lesson under controlled condi- 
tions. No one, however, reported followinMy^h a procedure; there were no efforts to ' 
validate lesson plans. 

Interviewees at Army schools reported that the TEC lessons- being produced to 
support individual training are subjected to a validation process. No TEC lessons were 
reported to be used in the courses studied, and consequently no TEC development efforts 
were reviewed, but at least one instance of such validation was .briefly observed. 
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Internal Evaluation 

Internal evaluation determines how well instruction works during actual training with 
actual trainees. Validation occurs only once, while internal evaluation is continuous. The 
principal measure, is trainee performance on objective-referenced achievement tests. Anal- 
ysis qr evaluation of how the ISD process itself was earned out is also part of internal 
evaluation, and is sometimes termed process evaluation. 
V ; . 

' * 1 . Table 28 

i 4 - ■ 

T Internal Evaluation i % 

Incidence of Prerequisites and Procedures in Courses Examined (N-57) 



Prereqursites 


Present 


Absent 


Not Applicable/ 
Not Determined 


• • 

1 . Records of students' performance on 
achievement tests are available. 


28 


1 


28 


2. Ddcurnentstton is available of what 
occurred during the ISD process, 
including rationales for decisions, 
departures from standard procedures, etc. 


• 

T3 


19 


25 


Procedures 


t 

Don. 


Not Qpne 


Not Applicable/ 
Not Determined 


1 . Specify evaluation criteria (number and 
percent of persons required to pass 
tests, etc J. 


4 


28 


- * 

25 


2. Identify the causes of shortcomings in the * 
' instruction and specify revisions. 


15 


* 

18 


24 



Efforts to investigate shortcomings in ongoing instruction and recon mend remedies 
were reported in 16 courses. Such efforts rarely involved comparing trainee performance 
to a standard of course adequacy; despite the almost universal availability of achievement 
test scores, a criterion of course adequacy was established in only four cases. Possibly 
trainers see little need for such standards in light of established limits on course attrition. 
If a requirement for course completion is to achieve every objective, for example, and 
if permissible course attritibn rates .are determined by higher command, it may be felt 
that no other standards are necessary. 

Some documentation of the ISD process itself was available in 13 courses. The com- 
prehensiveness of such documentation varied greatly. In some cases records were available 
showing when different activities had been started and completed, but in most instances 
documentation was less detailed. Notably absent from nearly all documentation were 
rattonalefr for the selection of tasks, setting, methods, and media* 

, ■ 4 
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^External Evaluation 

* ■• -• • 

Trainee proficiency and the adequacy of the instructional design process are deter- 
mined by using a standard external to. the course: v the performance of the graduate on 
'the job. The most accurate methods of external evaluation are direct observation and 
testing' of graduates. Reliance, however, must usually be placed on supervisors' summary 
evaluations (ratings) of performance.' Evaluation- information of whatever type should be 
obtained at a task level of specificity. More general evaluations are of little use in iso- 
lating the causes of. inadequate performance.. < 

Table 29 



External Evaluation: 
Incidence of Prerequisites and Procedures in Courses Examined (N-57)* 



Prerequisites 


Present 


— — \ 

_* Abtent 


Not Applicable/ 
Not Determined 


1 . Access to Supervisor* and job incumbents 
is available soon after arrival of graduates 
#t the job 

•r 


24 


• 


32 


Procedures 


Done 


, Not Done 


Not Applicable/ 
Nov Determined 


1. Construct evaluation instruments (mail 
questionnaires, job sample tests, inter- 
view guides, etc.). 


is 


.15 


23; 


2. Collect evaluation information. 


ie 


• 16 - , 


26 


3. Analyse data, identify cayses of 
deficiencies, and specify revisions. 


8 ; 


17 


* 

32 



Information about graduates 9 performance in the field was collected in 16 courses. 
In half of these cases, the inforpiation waT&nalyzed and used in making some revision in 
the course. Questionnaires to graduates andf supervisors— usually, but ndfc always, written 
at the task level— were the main method employed. In one instance a pre-existing Job 
Performance Measure was used. . , . " i. , 

In most of these cases the persons responsible for course development also planped 
and conduct6d*the external evaluation. In two courses, the Training Evaluation Division 
of an Air Force technical training school conducted .the evaluation and issued a formal 
Training Evaluation Report. Formal evaluations of the same kind were conducted for 
some Army , courses as well, by the school Directorate of Evaluation. Although no exter- 
nal evaluations had been completed for any of the Army courses studied, two were 
scheduled to begin in thfc near future. 

In many of the cases in which graduates' opinions or information about their perform- 
ance was collected, however, the effort did not appear to have a clear connection with the 
instructional design/revision process. Some interviewees reported that data collected from 
graduates or supervisors had never been made available to those who had designed the course. 
In other cases periodic surveys were conducted, as directives required, but with no apparent 
idea of how the survey responses would be used to determine the need to revise instruction. 
In general, whether personnel could perform on the job the tasks for which they had been 
trained was not determined. The ISD "closed loop" remained open. 
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SUMMATION 

The great majority of ISD steps are either rarely undertaken in the Services or are 
carried outjfi a manner which fails to meet the requirements of ISD. Application of 
ISD in any real sense is virtually restricted to five steps: Identification of Job Require- 
ments, Analysis of Tasks, Development of Objectives and Objectives Hierarchies, Devel- 
opment of Achievement* Tests; and External Evaluation. These^ all steps fbt which 
procedures are available, and which do not involve considerations of total system per- 
formance or training subsystem efficiency. 

The execution of even these steps, however, often falls short of , achieving the intent 
and purpose of ISD. Job requirements are often defined simultaneously with, rather than 
prior to, the determination- of training requirements, thus inviting .training considerations 
to influence job analysis; the means by which job-requirements then become training 
requirements are generally not explicit. Task analysis does not Usually includdKhe delinea- 
tion of task conditions and standards; on which the derivation of draining conditions and . 
standards would logically depend. Objectives are as likely to be*deyeloped from instruc- 
tional .content, as content from objectives; objectives hierarchies perje are rarely developed, 
with the result 'that necessary basic skills and knowledge are usually not incorporated in 
training design specifications. Tests are often derived from instructional content rather 
than directly from objectives, raising -the possibility that such teste may fail to measure 
attainment of the actual objectives, information collected about graduates' job perform- 
ance is seldom used to revise training. 

The potential of ISD to insure that training meets job requirements depends both 
upon (a) maintaining .the interdependency of steps in a derivative process originating 
■in description of the job itself, and (b) testing and revising the* products of each of these 
steps until they meet previously identified requirements. ' In practice>however, the nec- 
essary close connection between steps is generally not maintained, nor is the cycle of 
testing and revision carried through. 

The principal effect of the current emphasis on ISD appears to be an awareness of 
contemporary training methods and techniques. The use of behavioral objectives to ' 
describe training content and the specification of criteria of mastery for training objectives - 
is almost universal; and there is widespread awareness of the desirability of restricting 
training to what is needed for job performance. Although'the ISD cycle of evaltiation 
and revision is hardly ever carried through successive iterations, there is^widespread accept- 
ance of the principle that trainees' performance is the valid measure of training effective- 
ness. Although instructional activities, sequence, methods, and media are still determined 
by ud hoc personal judgment rather than systematic procedures, the conversion of tradi- 
tional group-paced training to individualized self -paced instruction, and the replacement 
of the lecture or tutorial with other media, are common occurrences in ISD efforts. 
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Chapter 6 
FINDINGS AND RECOMMENDATIONS 

The present study sought answers to these questions: 
V V • Do current methodologies as represented in guidance documents provide 

»* the means for .attaining the goals of Instructional System Development? 

- • Do applications of ISD reflect these goals? 
* How can ISD methodologies and applications be made more effective? 
The basic information for answering these questions was sought by analyzing guidance 
• for 19 individual steps of ISD in four Service Models, and the manner in which each step 
was carried out in 57 course design efforts. This approach yielded findings about meth- 
odologies (Chapter 2) and their applications (Chapters 4 and 5) specific to particular 
steps, as well as more general findings about the system. In this chapter findings and 
accompanying recommendation* for specific steps are presented first. They are followed / 
by a discussion of the general adequacy of ISD and recommendations for its future 
implementation. 



FINDINGS AND RECOMMENDATIONS FOR 

SPECIFIC ISD STEPS ' 

Need /Discrepancy Analysis * 

Findings \ _ „ 

1. Methodology • Present ISD n.^deU do not include procedures for identifying problems 
in existing training and for identifying a need to undertake ISD. Guidance is needed ' 
on how to identify discrepancies between existing training and the field requirements 
for a job, and how to revise graining short of undertaking the entire ISD process. 

2. Application - ISD is not generally initiated in response to specific discrepancies /*> 
■> between training and field requirements. 

3. Application "- ISD iirfuually initiated in response to a directed change (e.g., 
provide individualized and self-paced instruction) or to a requirement to revise 
existing training in accordance with ISD methods. 

Recommendations 

1. Where existing training is being examined, ISD methodologies should emphasize 
evaluating and improving tha training, rather than simply assuming that development * 
of a new course is appropriate. At present new courses are rarely evaluated any 

more rigorously than the ones they replace. 

2. To increase the Emphasis on evaluation and improvement, specific procedures should 
be developed both to identify faults in existing training and to determine efficient 
boundaries for the ISD process. 
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IDENTIFICATION OF JOB REQUIREMENTS 
Findings » . . o 

1 . Methodology - Current guidance for identifying Job requirements permits con- 
siderable latitude in thettoroach taken and the level of description .used, with 
resulting variation in the reliability and utility of the information developed.. 

2. Application - Job analysis is usually confounded with the selection of tasks for 
training. Emphasis U not given to independently specifying requirements as they 
«xi$t in the job. 

3. Application - Job-task lists from occupational surveys (e.g., CODAP, NOTAP) are 
sometimes available. The information provided by such lists, however, is often in 
part about classes of activities rather than aboiit tasks, and to that extent may not 
serve as an adequate base for deriving training. Often, complete task lists are not 
developed.' ' 

- « t. 

Recommendation 

1. Training developers should be required to provide and maintain a description of job 
task requirements distinct 'from a listing of tasks selected for training. This would ' 
make explicit the extent to whict training requirements differ from job requirements. 

Selection of Tasks for Training 

Findingi 

1.. Methodology - There are no measures of system effectiveness that can be used to 
validate criteria for selecting tasks for training, and rides for applying such criteria. 
As a result, the choice of criteria to be used in selecting tasks to be trained is left - 
to personal judgment. 

* \ * 

2. Application - Selecting tasks for training is generally not preceded by a separate 
and distinct delineation of the tasks required -by the job. 

3. Application - Task selection for training is usually not done systematically. Its 
rationale is rarely explicit. 

4. Application - Task priorities (that would provide the basis for getting the maximum 
training benefits from the available funding) are not specified. 

Recommendations v 

1. Training developers should be required to make explicit the basis on which they 
select tasks for training, and to specify priorities among both the tasks selected and 
the tasks rejected for training for a particular job. 

2. In the absence of information about the effects of task selection criteria on system 
performance, guidance should be developed on the types of task priority information 
likely to bef relevant for different classes of jobs. 
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Analysis of Tasks 

Findings 

1. Methodology - Procedures for analyzing tasks are adequate. 

2. Application - Job task conditions and standards, as distinct from training conditions 
and standards, are seldom identified. Course control documents do not require that 

o job characteristics be specified. 

Recommendation 

1. Benefits of attempting to modifv current practices do not. appear great relative to 
- costs. No change in present practice is recommended. , 4 

Construction of Jot) Performance Measures 

Findings 

1. Methodology - Procedures for developing JPMs /are adequate in the ITRO model, ■ 
unclear in the Marine Corps model, and not included in the AFP 50-68 model. 

2. Application - JPMs have been developed in jobs where the consequences of inade- 
quate performance are especially serious, such as in flight training or use 6t special 
weapons 

3. Application » In only one instance were JPMs developed as part of ISD to validate 
within-course tests. 

Recommendation „ c^WL 

1. While theoretically worthwhile, JPMs are costly to develop. It is 3B*stic to 
« recommend their development except in special instances. No change in present 
practice is recommended. 

9 t 

• * C 

Selection of Setting 

Findings 

1. Methodology - A systematic procedure' for determining the optimal setting for * 
training does not exist. The development of such a procedure waits upon a means 
for measuring system performance, which' in turn will permit the validation of site 
selection criteria and decision rules. , 

2. Application - In general, training developers do not have the authority to designate 
and develop training in different settings. 

Recommendation 

1. The choice of training setting has effects on the operational system beyond the 

training subsystem. It is appropriate that the setting selection be made at a higher 
r level than training developer. In the absence of a means for assessing total system 
effectiveness, no change in present practice is recommended. 
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Development of Training , 
v Objective* and Objectives Hierarchies 

Findings 

1. Methodology - The ITRO model provide! the most comprehensive, explicit, and 
straightforward procedures for translating job requirements into training objectives. 
Deficiencies in the Marine Corps model could encourage writing' objectives to match 
what is being taught, rather than to meet job. requirements,. 

2. Application - The specification of training objectives is virtually universal, but the 
procedures used to identify objectives are highly variable and frequently unclear. 
There is evidence that objectives are often prepared after the fact and are derived 
from training content rather than used to generate it. 

3. Application - Terminal objectives are seldom subjected to an explicit hierarchical ' 
analysis to derive intermediate objectives. A determination of the skills and knowl- 
edge that would enable the trainee to meet the terminal objectives would emerge 
from such an analysis; these often are not identified. 

4. Application - Even where records are maintained, formats for displaying the relation 
between tasks and training objectives make it hard to determine what objectives 
have been derived from a given task, That is, tasks that represent objectives are 
displayed, rather than objectives that have been derived from each task. Thus, the 

.. Justification for training for specific objectives is» often not clear. 

Recommendation » 

1 * Jke derivation of training objectives from job tasks' should be made explicit in a 
format that cross-references objectives by t<uk. ' 
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t Development of Achievement Tests 

i • * x 

Findings 

♦ 

1. Methodology - All of the models require that achievement tests be developed from 
training objectives, rather than fipm the content of lesions, and all provide some 
information about test construction. All models lack procedures for maintaining 
congruence between the .behaviors implied in an objective and the actual require- 
ments imposed by test items (e.g., use performance tests to measure skilled behavior; 
require' that concepts be applied when an objective implies their use rather than 
their recall or recognition). ' 

» • «*■ ' 

2. Application - Many achievement tests are derived, not directly from training objec- 
tives, out from training content. Knowledge tests are particularly likely to be 
derived from content: In these cases, no independent criterion exists to deter- 
mine whether training objectives have been met. • 

3. Application - In general, little or no consideration is given to matching the type and 
level of test items to the behavioral requirements of objectives (see Finding 1). 

Recommendations » 

1. ISD models should explain and emphasize the purpose and need for deriving 
achievement tests from training objectives rather than from training content. » 

2. ISD models should be expanded to provide procedures for identifying and main- 
taining congruence between the behavioral requirements of objectives and test items. 

123 



Identification of Entry Behavior 



Findings «x 

li Methodology - The major ISD models provide procedures for adjusting training 
objectives to match trainee capabilities. 

2. Application - Estimates of capabilities of trainee populations are not verified by 
testing before the training is implemented. 

Recommendation ■ 

1. The advantages of correcting inaccurate estimates of trainee capabilities do not 
appear to justify the costs of the measurement that would be Required. Over- 
estimates are likely to be identified during the validation of instruction, and under- 
estimates usually become evident during the conduct of instruction* No change in 
present practice is recommended unless large investments in instructional materials 
are involved (e.g., Training Extension Course (TEC) development). 

Classification of Objectives and M 
Selection of Instructional Activities 

Findings 

1. * Methodology ; Procedures for classifying training objectives, and for selecting instruc- 

tional activities accordingly, are not highly developed. Different models use different 
taxonomies for classifying objectives, and guidance for both classifying objectives 
and selecting instructional activities is provided largely by example rather v than by 
means of explicit decision rules. 

2. Application - Training objectives generally are not classified, and instructional activ- 
ities generally are not specified. n 

Recommendation | 

1. Explicit decision rules for classifying objectives and selecting instructional activities, 
should be developed. In the absence of such rules, no change in present practice 
is recommended. 



Selection of Instructional Methods 

Findings 

1. Methodology - All models describe and discuss alternative instructional methods. 
They specify, prerequisite- conditions (e.g., setting, group size) for the use of particu- 
lar methods but provide little information about the relative effectiveness of different 
methods, either for particular types of content or for trainee populations. Though 
perhaps sufficient to allow the training developer to reject inappropriate methods, 
the information base and the models themselves are not sufficient to provide for 
selecting^ optimal methods. 
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2. Application - Training methods are not systematically selected either on the basis 
of instructional activities (which are also not specified) or on the basis of trainee 
characteristics. 

3. Application • Changes in training methods are almost always in response to com- 
mand policy. 

Rccofflffisndstion 

1. Present training technology is not advanced enough to .support proceduralized 
derivation of training methods from previously specified instructional activities 
and trainee characteristics. Information about optimal training" methods for dif- 
ferent training situations, and procedures to enable developers to identify the most 
promising methods, should be developed. In the absence of such procedures, no 
change in present practice is recommended. 



1. Methodology - Procedures for selecting training media appear adequate to match 

0 the presentation' (stimulus and response) requirements of instructional activities to 
appropriate media, if instructional activities have been specified in sufficient detail. 

2. Application • Training' media are not systematically selected on the basis of require 
ments of instructional activities (which are also not specified). 

3. Application • Developers generally do not have freedom to select among alternative 
media. Choices and changes in media are usually directed by command policy. 

* 

Recommendation 

1. Selection of appropriate media isvpontingent on how well instructional activities 
have been specified. No change in the current models for matching media to < 
activities is appropriate until activities are more widely specified, and these models 
can be tested. . .. ' 



1. Methodology • Universal principles for grouping and sequencing instructional objec- 
' tives do not exist, other than that dependent objectives be placed later than those 

on which they depend. Alternate theories and conflicting strategies abound, with 
no knowledge base for resolving them. Because systematically related principles 
for promoting learning are lacking, sequencing instruction must be left to the 

" individual judgment of the training developers. 

- » 

2. Application • Practices in grouping ani sequencing instruction generally give no con- 
sideration to learning requirements other than the order imposed by obvious depend- 
encies. Attention is given to constraints of non-learning factors such as equipment 
availability and scheduling. ' 



Selection of Media 



Findings 



Grouping and Sequencing of Instruction 



Findings 



Recommendation ^^Sfe^i 

1. Organised and tyitematically related principles of learning on which to bate grouping 
and sequencing decisions are not availablerrhe development of such information 
falls within the psychology of learning and is beyond the scope of recommendations 
in this report. In the absence of such information, no change in present practices 
is recommended. , ' v 



Development of Plan for 
Authoring and Managing Instruction 

Findings 

m 

1. . Methodology - All models require the preparation of a plan for authoring and 
managing instruction. 

2. Application - Plans that record course content are often prepared. 

3. Application - Plans typically do not specify instructional events and are rarely used 
to develop instructional materials. 

* 

Recommendation 

1. Explicit decision rules for selecting instructional activities do not presently exist 
(see Classification of Objectives and Selection of Instructional Activities above). 
Until such rules are available and generally acknowledged as valid, a requirement to 
specify instructional activities in an authoring and managing plan is likely to be 
viewed as a pointless exercise. No change in present practice is recommended. 

Review and Selection of Existing Materials 
Findings * 

— i . / 

1. Methodology - The ITRO and Marine Corps models statelhat^deeisiolns to use 
existing materials (rather than author new instruction) are' to be based on the 
appropriateness of these materials to the previously specified characteristics of 
objectives, methods, and media. The AFP 50-58 model does not provide guidance 
for reviewing ana selecting existing material. 

2. Application - Characteristics to be identified in judging the appropriateness of 
existing training materials are not specified. 

3. Application - Review md use of existing training materials is minimal, except for 
those in a course that is being revised. 

Recommendation 

1. , The specification of necessary properties of materials for particular training situa- ' 
tions, and the description and cataloging of existing materials to permit the inter- 
change of matching components across courses, represent a degree of perfection 



that is not presently attainable. An attempt to reduce the review and selection of 
existing materials to a systematic procedure it to act at if the methods of a well- 
developed techndlogy were available in an area in which judgment it in fact the 
dominant factor. No change in present practice is recommended. 

. . Authoring of Instruction 

Fmdtogt 

1. Methodology - All models specify that instructional materials undergo tryout during 
the tuthoring process. t The ITRO and Marine Corps models emphasize that instruc- 
tion should be lean to insure the economies of minimal instruction. 

2. Application • There is little awareness of the concept ot lean instruction, and 
few attempts to develop it. 

3. Application - Instruction is rarely given tryout and revision during authoring. 
Recommendation . 

1. Training managers should receive guidance on the purpose and importance of 
developing lean instruction. Guidance should indicate the role of tryout and 
revision of instruction as a necessary element of this strategy. 

Validation of Ihrtruction 

Findings 

1. Methodology - All models specify satisfactory procedures for validating instruction. 
The adequacy of training materials for attaining objectives is verified through the 
administration of 'achievement tests. • 

2. Application - Validation criteria— that is, evidence that instruction is satisfactory- 
are rarely specified. 

3. Application • Instruction is rarely validated before it is implemented. When vali- 
dation does occur, it is training materials (e^., textbooks, tape/slide programs) 
that are evaluated; instructor lessons and lesson plans are almost never evaluated. 

Recommendation 

1. Of the three major types of ISD evaluation (vaUdation, internal evaluation, external 
evaluation), validation has the greatest potentiator effecting improvements in 
instruction. Once instructional materials have been produced and instruction has 
been implemented, changes are less likely to be introduced, and new materials are 
more difficult to generate. Validation trials to meet specified criteria should be 
required before new instruction it approved. 
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Internal Evaluation 

Findings • • • ft . , • • ♦ 

■ * . . ■• , • ' 

1. Methodology - All models specify adequate procedures for the internal evaluation 

of training. Quality, control of the training product is to be maintained through 
the administration o( objective-referenced achievement tests. 

2. Application - Evaluation and revision of instruction based on needs revealed in 
student performance (product evaluation) are generally not done. 

3. Application - Training design decisions are rarely documented (process evaluation), 
to facilitate redesign when instruction is found to be inadequate. ' * 

m <• . . * 

Recommendation 

" .* v> 

1. Trainers should be required to determine and record trainee performance for each 
objective. Although absolute standards to identify when training revision is needed 
are difficult to establish, the recording of specific trainee performance would pro- 
vide a desirable prerequisite to any revision. Moreover, it would suggest relative 
standards for the need to revise training. 



External Evaluation 

Findings 

1. Methodology - the KFRO model prescribes the most reliable and^nost costly way 
to measure the adequacy of the instructional design process: administering Job 
Performance Measures to graduates in the field. It also provides the most guidance 
for isolating causes of performance discrepancies after the external evaluation of 
training. If summary -evaluations are to be used, the ITRO and AFP 50-58 models 
specify that information be obtained at the task level of specificity, while the 
Marine Corps model does not. 

2. Methodology - None of the models specify criteria that should be used to determine 

whether training is to be revised as a result of external evaluation, nor how to 

arrive at such criteria. None tell how good job performance must be to indicate 

that training is acceptable. 
»* c ■ ■ ■ « 

3. Application • The effectiveness of training is virtually never evaluated by the admin- 
istration of Job Performance Measures to job incumbents. 

4. A pplication - Supervisor summary evaluations of job incumbent performance are 
occasionally obtained but usually are not provided at a task qr training objective 
level of specificity. Even when performance and job requirements information is 
obtained, it is rarely used to redesign training. 

Recommendations % 

1. The high cost of administering performance tests to job incumbents and the diffi- 
culty of maintaining the necessary degree of objectivity and standardization pre- 
clude their widespread use to evaluate training. No change in present practice is 
recommended. 



3. Training eta be evaluated <by gatherinf information (rather than direct measurement) 
■bout Job performance and Job requirements, at a task and training objective level 
of specificity, far more thoroughly than is presently done. The failure to obtain 
and use such information Is a major shortcoming in current applications of I8D. 
Whim decision rules for using such information cannot at present be based on 
m eas ur e s of total system ef f ec tiv e ness , other jneens for arriving at such criteria 
•'• are available. It is recommended that operational commands define both the 
specificity of the task description and the level of performance they would be 
willing to use to evaluate the acceptability of job incumbents. These are the 
criteria that supervisors should use to judge (rate) field performance and establish 
the need for training revision. 




GENERAL FINDINGS AND RECOMMENDATIONS 



Judgments about the adequacy of Instructional System Development, its representa- 
tion in the Service models,' and the way Jt is currently being applied depend on one's 
conception of what ISD is and what it is expected to .accomplish. * • ' 

The most general way to define ISD is as a means of designing training to optimise 
total system effectiveness. Criteria do not exist for measuring total system effectiveness; 
► ISD is not being used to achieve, nor can it be expected' to result in, optimising the 
effectiveness of the total system. 

A more circumscribed view of ISD is as a methodology for maximising training " 
efficiency within the training subsystem. However, information about the effectiveness 
and costs of different training strategies is far from complete, and a trial-and-error 
approach to maximising efficiency is not practical', given the number of possible combi- 
* nations of methods. While ISD does provide a framework for comparing alternative 
training strategies, it is not currently being used, nor can it be regarded, as a methodology 
for maximising training efficiency. 

A more highly focused view is that ISD is a methodology for insuring that training 
is relevant to the job. Its iterative and derivative character virtually assures that training 
will- be relevant if available procedures are faithfully carried out. In practice, however, 
many of its components are omitted, and the 'dose connection between components that 
is essential to make the process truly derivative is not maintained. Most important, the 
testing and revision necessary to insure job relevance generally do not occur. The potential 
of ISD to insure that training meets job requirements is not being realised. 

A final conception of ISD is that it is synonymous with the use^of modern training 
technology. Any of the steps in the training development process are a part of modern 
training technology, and so are any of those particular training and evaluation methods 
currently being emphasised, {e.g., self-rmccdf instruction, computer-managed instruction, 
criterion-referenced testing). This definition of ISD is clearly the least demanding, since 
in essence it holds that 'undertaking any training development step or using any such 
training or evaluation method constitutes ISD. It is the defin'tion that is most adequately 
represented in current applications of ISD. 

In sumnvury,' then, two effects of tsD are currently possible: insuring that training 
meets Job requirements and promoting the use of modem training technology. The former, 
which Is dearly the more desirable, is not being achieved. The I8D modes does provide 
the methodology for. making training relevant, but the mere existence of the model does 
not compel trainers to follow it Trainers are relatively free, within fairly broad limits, 



to determine the extent Jko which they wili conform to the I8D process snd actually uie 
its product* m designing training. For example, front-end analysis may b* undertaken, 
but its reaulte can ba and frequently ate ignored. ' Training objectives often are devel- ' 
oped with indifferent* to,, or to ignorance of, actual teak requirement!. Many tests and 
instructional materials are developed without regard to training objectives. Inforastieff 
about the performance of trainees, once collected, Is often not used to reviae training, 
and feedback about graduates, if initiated, is often not acted upon. ~ 

Because I8D is a process, it is difficult to observe directly . The occurrence <pf a 
process is generally inferred from the presence or absence of its products, in the case 
of ISO, however, the mare existence of its products-Job task lists, training bbjec'avws. 
achievement tests, and the like-does not by itself indicate that may have been used fii 
training development. Considerable, evidence that many ISD products remain unused 
_ leads to the conclusion; that, while the generation of these product* can ba mandated, 
the ISD process and the appropriate use of the* products during training development 
cannot. . . 

Similarly, a routine allocation orresponsibiiities in I8D does not necessarily guar- 
antee that these responsibilities will be recognised, accepted, and carried out. For example, 
ISD methodology requires developers to specify the way in which proficiency wUl be 
developed or mediated fofr all job tasksHminedtetely, through 

through advanced training or job experience; or through direct support of job performance 
by procedural aide or other means. Yet this requirement is frequently ignored. ISD is v 
generally conducted to develop training for only one particular setting, and the manner in . 
which skill will be acquired for teaks that are not selected for training in that setting js 
usually not specified. Even where skills were explicitly identified for later acquisition, 
the present study found little evidence either that means were developed for subsequent 
training or that operational unite were informed of their responsibilities for insuring that . 
these skills be acquired.. (This observation is based on training development within the 
training subsystem. The present study did not examine training devefcned or conducted 
in operational units. Possibly such an examination would indicate that training for job 
thills in these units is being conducted in a more comprehensive manner thin was sug- 
gested by the evidence in the present study.) 

The current failure of ISD applications to insure that training meets job requirements, 
then, is largely due not to inadequacies in the methodology, but to omissions and to 
failure to use its products in a way that makes the process truly derivative. An implica- 
tion of these findings is that future efforts to implement ISD should concentrate on 
finding ways to maintain the integrity of the modal. ' at 



The findings of the study do not of themselves indicate how to assure rigoroaw 
adherence to what is clearly a very demanding model, but they do suggest that intro- 
ducing changes solely within the training subsystem is not likely to have any great effect. 
The data strongly suggest the need for checks and balances, to guard against omissions in 
the. ISD process and failures to use the ISD products. It would appear logical, therefore, 
to provide for an expanded role by operational commands-the party directly affected 
by shortcomings in training, and best able to assess the effects of training. 

Such an expanded role for operational commands U, in fact, implied by the model 
itself. The derivative and iterative aspects of ISD depend on feedback and exchange of 
information between trainers and users. It is difficult to see how ISD could be more 
rigorously applied uiuess such an exchange takes place. In principle; the training sub- 
system seeks information about field requirements and performance as the foundation on 
which training is constructed. In practice, however, the study found that the training 
subsystem, does not have this basic orientation, often giving insufficient .attention to the 
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effective um of this information. It is reasonable to suggest that a balanced relationship- 
one that fosters active participation by, and* communication between, operational and ' 
training commands-is essential to the ISD process. i 

the following recommendations define the meant by which operational commands 
can assume a greater role. Under these recommendations, operational commands would 
participate to the greatest extant in those parts of ISD where job performance is repre- 
sented and where Job performance requirements are translated into training requirements. 
Thorn parts of ISD concerned with the design of Instructional strategies to meet train- 
ing and job requirements would remain the province of the training community. 

It is recommended: 

(1) That job requirement*, (skills and knowledge required for successful job 

t performance) he jointly defined, and agreed to*£y training and opera- ' 
tiondl command*. This recommendation is a prerequisite to Recommenda- 
tion 2, and to all subsequent training development activities. If successful 
dialogue, negotiation, and agreement are to follow, training requirements 
must be bated on a mutually agreed-upon definition of job* requirements. 

(2) That training requirement* (skills and knowledge to be available at the 
conclusion of training) oe jointly defined, and agreed to, by training 

• and operational commando. * Such an agreement should identify the specific 
tasks and standards to which proficiency will be developed, and should 
delineate the respective responsibilities of the two parties. This includes 
providing a means for bringing job incumbents to the desired level of 
proficiency whenever agreed-upon training requirements do.not match job 
requirements. * " 

(3) That operational commando be required to evaluati the performance of 
training graduate* and report their finding* to the training command*. 
Unless operational commands evaluate performance, feedback from users 
to trainers will not have astound basis. Without reliable information about 
the effects of training, specification of training requirements will not 
serve its purpose. 

(4) To implement Recommendation 3, that x ta*k-*pecific criteria for evaluating 
the performance of graduate*, including method* and etandard* to be 
employed, be Jointly defined, and agreed to, by training, and operational 

■ command*. Evaluation criteria should be at the task level of specificity 
to permit clear and usefuj diagnosis of training. More general evaluations 
- are of little use in isolating the causes of inadequate performance. Eval- * J 
uation criteria must be jointly agreed to if the results of evaluations are 
to be accepted as valid, and acted upon. 

Giving the operational commands a greater role in both establishing training require- 
ments and determining whether requirements have been met will not of itself guarantee 
that ISD procedures will be vigorously applied to the development of training. It will, 
however, increase the involvement of those who have the most fundamental interest in 
seeing that training has been adequately designed and conducted. 

So. long as training development and evaluation are regarded as a separate activity 
of the training commands, there is little reason to expect that ISD will be applied any 
more effectively than under the present conditions. If training and operational commands 
share these activities— each making its specialized contribution to complement the work 
of the other— the potential of the ISD process for improving training will be enhanced. 
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ISD ACTIVITIES QUESTIONNAIRE 
(MAIL SURVEY INSTRUMENT) 
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ISO ACTIVITIES QUESTIONNAIRE 



You are being asked to complete this qi 



to provide information about courses/ 



training program* where Instructional System Development (ISD) procedures nave been 



of a study of ISD In the Armed 
mstaUattons to obtain more 



applied. This information will be used in-the first 
Forces. In a later phase/visits will be made to 
detailed information. ' . 

In addition to identifying problems that have ariien ta applying ISD, the study seeks to 
identify those factors associated with both successful and unsuccessful application*. This 
information will be used to determine w|at organisat onal and other support is required for 
I8D implementation. v - ' 



1. 



Instructions for Completing the Questionnaire 

Enter identification information below. 



Organisation: 
Address: 



(Command or 



School or Unit) 



Person to be contacted for additional information 

« 

Commercial phone: " 



Enter below the total number of courses/training programs administered by the above 
organisation. Pat the purpose of this questionnaire, "courses/training programs'* include 
any course or program of instruction, training, or education. 

.. . • . , 

Total number conducted at the school or unit: - 



Total number conducted by correspondence or extension: 



3. On the following page, list the titles of all courses/training programs for which \ob analysis 
data (task lists] have been compiled -and are currently available. Although this questionnaire 

v does not require ISD to i* defined in terms of any particular set or sequence of actions, 
it does require that a list of job tasks be available. Therefore, list only courses/training ' 
programs for which task lists are available. 



For each course/training program that has been listed, provide information about ISD 
activities by completing columns 24%, on the following pages. 

• — — * 

Human Resources Research Organization 
^(Contract 903-77-C-033, 

Office of Assistant Secretary of Defame 
. (Manpower, Reserve Affairs, ft Logistics) 
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Instruction* for Comptotyfc Columns 1 • 23 

Column 1 : Enter Courses/Training Programs for which job analysis data (task tots) art avp liable. 

Column 2: Emar N jf%>o job erietysis data Ware/are being used to develop a now oourao. 

Enter E if tha Job analysis data ware/art tying used to evaluate/rev las an existing course.. 
Enter O if the job analysis data were/ere being used for other purposes than course development 

r- or revision. , 

Column 3: \ If a couraa was/hWig developed or revised, enter tha approximate date whan tha first graduate* 
Vaaehed/wlll reach the fob. 

Columns 4- Provide information about ISO activities ^ - 

22: Enter A if ... the activity was undertaken, and documentation or products are available. 

Enter P if . . . the activity was attempted, but products are not available due to methodology 

or other problems. 
Enter N/C if . . .^he activity is being undertaken, but is not completed. 
. Enter. N/A if ... the activity is not applicable or was not undertaken. 



Approximate Bete when first graduate! 
reached (or wMf leach) the job. 



N ■ New Couraa 
E ■ Existing Courts- 
0 ■ Other Purposes) 



COURSES/ 
TRAINING PROGRAMS 

FOR WHIC 
JOB ANALYSIS 
ARE AVAILABLE 

0 ' 



OATAr 7 



Analyzed proMem/systsm/existing eourss to dstsrmins 
need far training development. 



Obtained priority Information obouttasfct (frequency, 
delay toleranos, critiesl||y, etc.) 



Ansryzsd tasks inte eNffcnts. conditioni, and 



Satacfrd ta*s for training pn tht 
of task priority information. 



Chntarad \mkt and ulactad inttruc- 
ttonal aattim on tha feaais of priority 
information, rasourcas, dnd output 
raquirtmantt. ■ . 



Otvalooad tarminal and inttr* 
madiata laarnim obi#cti% 



Saqutncad and 
oblactfvat. 

10 



ctwtarad 



/ 



/ 
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uvvfiofm, too qui, mn iwum inn. 



Identified kM«Mp and thills of entering treinees. 



11 



12 



-A 



Identified type<a) of teeming required for each objective, and specified corresponding 



13 



Identified medio appropriate to type of laaming and laorning activities. 



14 



Specified plan for pacing, instructor rola, group/indrvidutlfzed praaantation, 
ichaduiing, ontry/oiclt requirements, ate. 



IS 



Reviewed/selected appropriata axitting instruction materials. 



16 



Authorad/producod now instruction matarials. 



17 



Vr iidated instruction matarials. 



18 



Conducts*! inWuctiop. 



19 



Analyzed studant performance (for intamal 
evaluation of course). 



20 



Revised training according to results of 
intamal evaluation. 



21 



Determined if graduates ware meeting 
performance requirements on the job 
(external/field evaluation). 



22 



Revised training according to 
results of extemel evaluation. 



23 



Turn page to list 
additional courses. 
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INCIDENCE BY SERVICE OF PREREQUISITES AND 
PROCEDURES IN COURSES EXAMINED 
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Appendix B 



Incidtnot by Strviet of Prerequisites and 
Procedure* in Courses Examined' 



Present/ 
Accompliihed 



Absent/ 
Not Accomplished 



Not Applicebfe/ 
Not Determined 



Need/Discrepancy Analysis 

Prerequisites 

1 . Information it available about 
possible discrepancies between 
training and field requirements. 



Army 
Navy 

Marine Corps 
Air Force 



2 
11 
2 

16 



0 
1 
0 
1 



5 
11 
S 
3 



Procedures 

1 . Analyze initiel end supple- 
mentary information. 



2. Identify end specify 
discrepancy. 



Army 
Navy . 
Marine Corps 
Air Force 

Army 
Nevy 

Marine Corps 
Air Force 



2 
10 
2 

1.0 

2 
9 
1 
11 



5 
10 
2 
5 

5 
11 
.2 

5 



0 
3 
3 
5 

0 
3 
4 
4 



3. Specify ISO entry point 
end boundaries of redesign 
process. 



Army 
Navy 

Marine Corps 
Air Force 



^1 
8 
1 
9 



6 

12 
3 
6 



0 
3 
3 
5 



Mtfitif icstion of Job flM^sWtfYtftfitt 

Prtrequisitts 
Nor* 

Procedures 

1. Construct provisional task list. Army 7 0 0 

Navy 16 4 3 

Marine Corps 6 0 '1 

Air Force 9 9 2 

(Continued) 

1 Total number of courses examined in eactt Service: Army-?; Nevy -23, Marine Corps-7; Air Force-20. 
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Identification of Job Requirements 

(Continued) 



Prstsnt/ 
Accomplished 



AbMm/ 
Not Accomplished 



Not Applicable/ 
Not Determined 



2. Verify and revise on basis of 
review/survey of job 
incumbents. 



3. Colltct task priority 
information. 



Navy 

Marina Corps 
Air Force 

Army 
Navy 

Marina Corps 
Air Force 



6 
11 
4 
6 

5 
7 
4 
8 



1 
8 
1 

10 
1 

12 
0 
10 



0 
4 
2 
4 

1 
4 

3 
2 



Selection of Tasks for Training 

Praraquisitas 

1 . Tasks performed in the 
job are listed. 



Information is available . 
for establishing importance 
of tasksand need for* 
training. 

Decision rules to be applied 
to task information are 
available. 



.Procedures 



1 . Apply decision rules to 
information for each task 
to determine training 
priorities. 

2. Select tasks for training 
on basis of training 
priorities and resource 
availability. 



Army 
Navy 

Marine Corps 
Air Force 

Army 
Navy 

Marine Corps 
Air Force 

Army 
Navy 

Marine Corps 
Air Force 



Army 
Navy 

Marine Corps 
Air Force 

Army 
Navy 

Marine Corps 
Air Force 



7 

15 
7 
9 

5 
7 
2 
9 

3 
2 
1 
1 



2 
1 
1 
1 

2 
2 
1 
1 



0 
4 
0 
9 

2 
11 
1 
9 

4 

17 
2 
17 



5 

V. 
1 

17 

5 
17 
1 

17 



0 
4 

0 
2 

0 
5 
4 

2 

.0 
4 
4 
2 



0 
4 

5 
2 

0 
4 

5 
2 



(Continued) 
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S Prtwnt/ AbMnt/ Not Appliabit/ 

Agftffgjtehjjd , Not Accompttotd Not Dttormintd 
Analytic off Toaka " 



ProroquititM 



1 . Taskt satactad for training 


Army 


7 r 


0 


0 


ift list ad. 


navy 


10 


15 


2 




Marina Corps 


3 


2 


2 




Air Forca 


16 


3 


2 


Procaduras 


• 








1. For aach task spKif y for 








» 

* 


tha k>b mvlronmint * 










- Conditions of porformonct. 


Army 


7 


0 


0 




Navy 


^ 3 


1* 


5 




Marina Corps 


4 


0 


3 




Wr Forca 


2 


14 


4 

V 


- Behavioral tiomontt. 


Army 


6 


i 

0 


1 

f • 




Navy 


9 


9 


p 

5 


\ 


Marina Corps 


4 


o • 


3 


- 


Air Forca 

c 


12 


4 


4 


— Standards of Mrf ormanca 


my 


/ 


0 


0 




Navv 


o 


14 


4 


$ 


Marina£orps 


4 


0 


3 




Air Forca 


3 


14 

- 


3 


Construction of Job Parformanea 
Maasurat 










Praraqutsitss 






• • 




1 . Tasks lalartad for train inn 


nimy 




0 


3 


ara lift ad. 


navy 


3 


1 


19 


• 


iviarinv t*orps» 


1 


0 


6 




Air For c«i 




1 


i£ 

ID 


2. Training raquiramants for 


Army' 


3 


1 




thaaa tasks hava not baan 


Navy 


2 


2 


19 


idantifiad. 


Marina Corps 


1 


0 


6 


r 


Air Forca 


2 


3 


15 


Procaduras 










1. Construct a tast for maasuring 


Army 


4 


3 


0 


tha parformanea of aach task 


Navy 


2. 


17 


4 


satactad for training. 


Marina Corps 


0 


4 


3 




Air Forca 


5 


12 


3 



(Continued) 
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Construction off Job Performance 

i {Contintmf) 



PrtMnt/ AbMnt/ Not Applicable/ 

Accomplished Not Accomplished Not Pewrrnintd 



2. Validate each Job Parformanca 
Measure to insurt that it pre- 
dicta task parformanca. 



Army 
Navy 

Marina Corps 
Air Forca 



0 
2 

d 

1 



4 

17 
5 
12 



3 
4- 
2 
7 



Selection of Setting 

Praraquisites 

1 . tasks can ba assigned to 
any of savaral sattings. 



Army 
Navy 

Marina Corps 
Air Forca 



4 

8 
0 
6 



0 
10 

2 
11 



3 
5 
6 
3 



Information is available 
on costs of training in 
diffaram settings* 



Army 
Navy 

Marina Corps 
Air Forca 



0 
0 
0 
0 



0 
1 
0 
2 



7 
22 
7 

18 



Information is available 
on affacts of training in 
differ ant sattings on total 
system effectiveness. 



Army 
Navy 

Marine Corps 
Air Force 



0 
0 
0 
0 



0 
1 
0 
2 



7 
22 

•7 
.18 



Procedures 

1. Assign each task or 
group of tasks to its 
appropriate setting. . 



Army 
Navy 

Marine Corps 
Air Force 



3 
7 
0 
6 



4 

15 
6 
12 



0 
1 
1 
2 



Development of Training Objectives 
and Objectives Hierarchies 



Prerequisites 

1. Tasks selected for training 
are listed. 



2. Information is available 
about training constraints 
that make it necessary to 
modify task requirements. 



Army 
Navy 

Marine Corps 
Air Force 

Army 
Navy 

Marine Corps 
Air Forca 



7 
17 

5 
15 




0 
4 

0 

3 

0 
0 
0 
0 



0^ 
2 
2 
2 

7 - 
23 

7 
20 



(Continued) 
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Peasant/ 
Acocmptithad 



Abssnt/ 
Not Aoaamptishad 



Nat AppHcsb*/ 
Not DmrmiMd 



Dovoto0mofit of Training Otojaativas and 
Obje et lws Hierarchies tContimmd) 



3. Information it available about 


Army 


0 


<S 


"7 


how modification of task 


Navy 


0 


0 


23 


* •* 

raquiramantt will aff act 


Marina Corps 


0 


0 


7 


1 training aff iciancy. 


Air Forca 


0 


0 


20 


4. Estimataaof capabilitias 


Army 


0 


0 


7 


of antaring trainaat for I 


Navy 


6 


2 


16 


laarning and parforming 


) Marina Corps 


0 


0 


7 


aach task art availabla. 1 


A1r Forca 


0. 


0 


20 


Procadurat \ 




% 


• 




1 . Spacify talk raquiramants 


\*rmy 


0 

6 


0 


1 


(bahaviors, conditions. 


Nlvy 


22 


0 


1 


standards) for training. * 


Marina Corps 


5 


1 


1 


« 


Air Forca 




0 


2 


2. Parform hierarchical anal- 


Army 


3 


3 


1 


ysis of tasks to idantify 


Navy 


7 


16 


0 


intar mad iota training 


Marina Corps 


3 


' 3 


1 


objactivas. 


Air Forca 


1 


17 


2 



i/as ampvnam or MJiwavaniaiii i asxs 



Prerequisites 

1, Training objectives htvt 
bWh specified. 



Army 
Navy 

Marina Corps 
Air Forca . 



6 
21 

6 
18 



0 
0 
0 
0 



1 
2 
2 
2 



2. Instructional materials nave 
. not been developed. 



Army 
Navy 

Marine Corps 
Air Force 



3 
8 
2 
3 



3 
12 
1 

16' 



1 
3 
4 

2 



Procedures 

1. Determine appropriate > 
types of tests based on 
characteristics of objectives. 



Army 
Navy 

Marina Corps 
Air Forca 



0 
0 
0 
0 



3 
1 
2 
0 



4 

22 
6 
20 



2. Construct tests to < 
attainment of objectives. 



Army 
Navy 

Marine Corps 
Air Force 



6 
0 
4 

17 



(Continual) 



0 
18 
0 
0 



1 
6 

3 
3 



tie 



4 



PrtMnt/ AbMnt/ Not Aijfcctf*/ ' 

AocofnplifHsd Not Aoootwplishsd • Not DetSfrnlnsd 

Identifiaation of Entry Behavior . 

i ' 1 



1. Objectives have been derived 


Army 


3 


3 . • ' 


1 


through hierarchical anal 


Navy 


6 


15 


3 


\ ysitof tetkt. 


Marine Cor pt 


2 


4 


1 


/ 


Air Force 


1 


9 


3 


2. Teats art available to 


Army 




o- 


1 


. 9 measure objectives c 


Navy 


19 


0 


4 


* 


Marine Corps 


3 


0 


4 


> 


Air Force 


17 


0 


3 


Procedure! * 


'•» 






1. identify temple that it 


Army 


0 






reoreeentative of trainees 


Navy 


0 


20 


'3 




Marine Corps 


0 


6 * 


1 




Air Force 


0 


17 


3 


2. Administer tests to sam- * 


Army 


n 

w 


* 


i 

1 


pie, and determine accu- 


Navy 


0 


20 


3 


racy of earlier estimate of 


Marine Corps 


0 

* 


6 


1 


entry behavior. 


Air Force . 


0 


• 

17 


3 


* 

3, Add or delete objectives 


> * 

Army 


* 

0 . 


6 


1 


m , at indicated by teat 


Navy 


0 


20 


3 


results, and repeat cyck). 


Marine Corps 


o . 


6 


1 


« 


Ahr Force 


0 ' ' 


17 


3 


flaaslfhielinn of Obtoetkes and 






4 




Selection of Instructional Activities 


• 


V 






Prerequisites 


• 






■ 


1. Information it aval labia about 


Army 


0 


• 

. 0 


7 


types of instructions! activ- 


Navy 


0 


0 


23 


ities appropriate to acquiring 


Marina Corps 


0 


0 


7 


, different types of capabilities. 


r Air Force 


0 


0 

• 


20 


Procedure! 




r 






1. Clatsify eoch objective or 


Army 


c 2 


3 


2 


group of objectives accord- 


Navy 


4 


16 


3 


ing to type of capability. 


Marine Corps 


o 


6 


2 




Air Force 


2 ,4 


16 


2 


2. Specify instructional activ- 


Army 


0 


6 


V 


ate for each objective accord- 


Navy 


A 


10 


3 


ing to type of capability. 


Marine Corps 


0 


5 


2 


i 


Air Force 


0 


18 


2 



(Continued) 
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oil 

1. Siting hat b#*n tptcif M. 



Army 
Navy • 
Mtrtnt Corps 
Air Fortt 



AooompHihtd Not Accottipl I iht d 



Not Applicable/ 
Not Oatarminsd 



6 

22 
6 
18 



0 
0 
0 
0 



.J 
1 

2 



2. Trainot chaiw-toristtcs 
htvo boon idontifiod. 



Army 
Now ' 
Marino Corps 
Air For co 



0 
0 
0 
0 



0 
1 
0 
0 



7 

22 

7 

20 



3, Instructional tcttvtttos 
novo boon spocifiod. 



Army 
Navy 

Marino Corps 
Air For co 



0 
1 
0 
0 



6 

20 
6 
18 



1 
2 
2 
2 



4. Information it avoltoblo 
on how tho costs ond off oc- 
tivonon of sttomoto mothoda 
vary for spodftod aottlngs. 
troinoo choroctariotics t <and 
instructional octivitioa. 



Army 
Navy 

Marina Corps 
Air For co 



0 
0 
0 
0 



0 
1 
0 
2 



7 

22 
7 
18 



Procoduroa 

1. Spocify tho rootbods of Army 

instruction to bo omptoyod Navy 

for ooch objoctivo or group Marina Corps 

of objactivoa. Air Forco 



6 

22 
6 
18 



0 
0 
0 
0 



1 
1 
2 



tiitntinn of Madia 



Praroqulsltos 

1. Training dosignors oro 
froa to orioct from a 
rang* of madia. 



Army 
Navy 

Marina Corps 
'Air Forco 



0 
3 
0. 
3 



1 
6 
0 

3 



6 
16 

7 

14 



2. Instructional activities 
novo boon spocifiod. 



Army 
Navy 

Marina Corps 
Air Forco 



0 
4 
0 
0 



8 
18 

8 
18 



1 
3 
2 
2 
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3. Mo th o da of training novo 
boon spocifiod. 



lit 



Army 
Navy 

Marina Corps 
Air Forco ■■ 



3 
18 

0 
18 



(Continued) 



U 3 



0 

1 

0 
0 



4 
4 

7 

r 



Mtttlon of Media (Contkiutd) 

4. Information (• available ©on- 
eeroint the appropriatenef 
of different media to imple- 
ment different ectMtlei and 
to be uMd In conjunction 
with different methods. 

6. Information it sveilobl* con- 
cerning the costs of dif- 
ferent medio. 



Procedure! 

1. Determine which media 
will be euHabte to Imple- 
ment the instructional 
activities. 

2. Consider relative coats of 
media determined above end 
select most, economical set 
of madia. 



Army 
Navy 

Marina Corps 
Air Korea 



Army 
Navy 

Marine Corps 
Air Force 



Army 
Nevy 

Marine Corps 
Air Force 

Army 
Navy 

Marine Corps 
Air Force 



frseint/ 

^^COafl^piiS^iad 



0 
0 
0 



0 
0 
0 
0 



1 

2 
0 
0 

0 

1 

0 



Absent/ 
Not Aeoempliihsd 



0 
0 
0 
0 



0 
0 
0 
0 



6 
19 

6 
18 

6 

19 

18 



Not Applicable/ 
Not Of tautiffiad 



7 

?3 
7 
20 



7 
23 

7 
20 



2 
2 
2 

1 
3 
2 
2 



Qroupina, and t oeuoo s lna of 

I oeaitiAaLkA 

Praraqufrftas 

1. Knowtadga of tha tff acts on 
taarning of difftrant saquanc- 
ing plant iiavailabla* 



Procaduras 

1. Identify commonality of 
subjact mattar and antlcl- 
patad trantfar of laarning 
batwaan otyactivas. 

2. Idantify dagraa of dapand* 
ancy batwaan objactivai. 



3. Saiact ovarall taquancing 
principal). 



r 



Army 
Navy 

Marina Corps 
Air Forca 



Army 
Navy 

Marina Corp* 
Air Forca 

Army 
Navy 

Marina Corps 
Air Forca 

Army 
Navy 

Marina Corps . 
Air Forca 



(Continued) 



0 
0 
0 



1 

3 
0 
3 

2 
6 
1 
5 

2 
3 
2 
6 



6 

6 
18 



6 
16 

4' 
*14 

4 

13 
3 
9 

4 

16 

2 

10 



1 
4 

3 
3 

1 
6 
3 
6 

1 
4 
3 
4 
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r 



(ContkHmdt 



PftttOt/ 

Ac ooff >pii >h <d 



Abssm/ 
Not Ao oo w piWKWl 



Not ApplteeMs/ 



4. Group and sequence 
objectives. 



Army 
Navy 

Marin* Crop* 
AH* Force 



6 

21 
6 
18 



0 
0 
0 
0 



1 
2 
1 
2 



Dtv itopfMfit of Ptain fot Author ing 

i 



Prerequisites 

1 .tpb)octrv*s7 instructions! 
activities, methods, and 
media have all been specified. 



Procedures 

1. Specify the content and 
design of each lesson. 



Army 
Navy 

Marine Corps 
Air Force 



Army 
Navy 

Marine Corps 
Air Force 



0 
5 
0 
0 



3 
13 
1 

14 



6 
15 

6 
18 



2 
6 
3 
4 



1 
3 
2 
2 



2 
4 

3 
2* 



2. Specify now the instruction 
will be conducted and 
managed. 



A/my 
Navy 

Marine Cops 
Air Force 



8 
21 
6 

17 



0 
0 
0 
1 



1 
2 

•1 



Review and Selection of 
Existing Materials 

Prerequisites 

1. Instructional activities have 
been specified. 



2. Methods of training have 
been specified. 



Army 
Navy 

Marine Corps 
Air Force 

Army 
Navy 

Merine.Corps 
Air Force 



0 
5 
0 
0 

S 
20 

6 
18 



5 
15 

5 
18 

0 
0 
0 
0 



2 
3 
2 
2 

2 

3* 
1 
2 



3. Media have been specified. 



Army 
Navy 

Marine Corps 
Air Force 



3 * 
10 
2 

16 



2 
3 
0 
2 



2 
4 

5 
2 
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Pie ss o t/ 



AbMnt/ 
Not Aococwpmhuci 



Not AppHesMe/ 
Not OstsroMnsd 



Of 

(Conifnotd) 



Procedures 

M. 



Examine existing instructional 
materials to dttormint whether 
eny moot tho specif icet ions 
for inttructionti activities, 
methods, ond modio. 



2. Select materials or parts 
of motorists that moot tho 
specifications, or which i 
bo efficiently i 




Authoring of 

Proroquisitos 

1. Objoc t tvo s hove boon 
grouped end sequenced. 



2. Instructions! activities 
neve boon specified. 



3. Methods of training hove 
:ifiod. 



4. Medio hove been specif iod. 



5. Lesson structure ond 
content hove boon planned. 



Procedures 

1. Develop kin instruction. 



Army 
Navy 

Marina Corps 
Air Force 



Army 
Navy 

Marine Corps 
Air Force 



Army 
Navy 
Marina Crops 
Air Force 

Army 
Navy 

Marine Corps 
Air Force 

Army 
Navy 

Marine Corps 
Air Force 

Army 
Navy 

Marina Corps 
Air Fqrca 

Army , 
Navy 

Marine Corps 
Air Force 



Army 
Nevy 

Marine Corps 
Air Force 



1 
S 
0 
4 



1 
8 
0 
4 



6 
21 

6 
17 

0 
6 
0 
0 

6 

20 
6 

17 

3 
16 
2 

16 

2 
14 
1 

16 



1 
9 
0 
1 



4 

14 

5 
13 



4 

13 
6 
13 



0 
0 
0 
0 

5 
16 

6 
17 

0 
0 
0 
0 

2 
2 
3 
1 

2 
4 

' 4 
1 



3 
10 

J 
14 



2 
4 

2 
3 



2 
4 

2 
3 



2 
2 
1 

3 

2 
3 
2 
3 

2 
3 
1 
3 

2 
6 
2 
3 

3 
6 
2 
3 



3 
4 
6 
6 
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'46 



Authoring of Instruc ti on (Continued) 

0 

2. Try out instruction on small 
number of persons representa- 
tive of students. 



Present/ • AbMnt/ . °Not Applicable/ 

Aecompiilhed Not Accomplished Not Determined 



Army 
Navy 

Marina Corps 
Air For pa 



2 
7 

.0 
1 



3 
12 

2 
14 



2 
4 

5 
5 



3. Revise and augmant instruc- 
tion at i 



Validation of Instruction 

Prtrtqultjtg 

1. ObjactTva-rafarancad 
achiavamant tasts art 
available 



Army 
Navy 

Marina Corps 
Air Forca 



Army 
Navy 

Marina Corps 
Air Forca 



1 
3 
0 
1 



6 
20 
6 

16 



3 
11 

2 
14 



0 
0 
0 
0 



3 
9 

5 



2 
3 
1 



Procadurts 

1. Spacify achiavamant tatt 
A validation criteria (numbar 

and par cant of parsons in 
validation sampia raqyirad 
to pass tasts). 

2. Spacify additional valida- 
tion critaria. 



Prasant instruct^, adminis- 
tar achiavamant tasts, analyze 
rasutts, raviaa instruction, and 
rapaat cycla until validation - 
critaria ara mat. 



Army 
Navy 

Marina Corps 
Air Forca 



Army 

n 

Navy . 
Marina Corps 
Air Forca 

Army 
Navy 

Marina Corps 
Air Forca 



0 
7 
0 
0 



0 
4 

0 
0 

2 
7 
0 
0 



3 
0 
3 
16 



3 
11 

3 
16 

2 
11 

3 
15 



4 

7 
4 

6 



4 
8 
4 

6 

3 
5 
4. 

,5 



S 



Internal Evaluation 

Praraquisitas 

1 . Records of stud ants' perform- 
ance on achievement tests are 
available , 



Army 
Navy * 
Marine Corps 
Air Force 



3 
9 

3 
13 



0 
0 
0 
1 



4 

14 
4 

6 
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U7 



(Continuri) 



Z Documentation it available of 
what o<jcurred during the ISO 
proem, inducting rationales 
for decisions, departures from 
standard procedures, ate. 

Procedural 

1. Specif y evaluation criteria 
(number end percent of per* 
tone required to peat tests,, 
etc.). 



2. 



Identify the causes of 
shortcomings in the instruc- 
tion end specify revisions. 



Arfny 
Navy 

Marine Corps 
Air Force 



Army 
Navy 

Marine Corps 
Air Force 

Army 
Nevy 

Merine 'Corps 
Air Forge 



Present/ 
Accomplished 



5 

8:~> 



0 
1 
0 
3 

1 
5 
0 
9 



Abstnt/ 
Not Accomplished 



3 
6 

3 
7 



3 
10 
3 

12. 

2 
7 
3 
6 



Not Applicable/ 
Not Determined 



. 4 
12 
4 
5 



4 

12 
4 
5 

4 

11 
4 

S 



External Evaluation 

Prerequisites 

1. Access to supervisors end 
job incumbents is possible 
soon after arrival of grad- 
uates on the job. 

Procedures 



1 . Construct evaluation instru- 
ments (moil questionnaires, 
job sample testa, interview 
guides, e^cj. 

2. Collect evaluation 
information. 



3. Analyze data, identify 
causes of defictences, 
end specify revisions. 



Army 
Navy 

Marine Corps 
Air Force 



Army 
Navy 

Marin* Corps 
Air Force 

Army 
"Navy 

Marine Cotps 
Air Force 

Army 
Navy 

Marine Corps 
Air Force 



3 
8 
3 
10 



2 
8 
3 
6 

0 
7 
3 
6 

0 
3 
1 
4 



0 

w 

3 
5 
0 
7 

3 
S 
0 
7 

2 
7 
1 
7 



4 

16 
4 
9 



2 
10 
4 

7 

4 

11 
4 

7 

• 

5 
13 
5 
9 
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